The tooling I have been building this week reads a game's replay files. A replay is not video: it is the game's instructions for re-simulating a match, about 1.2 MB where the same match as footage is over a gigabyte, and its header carries the frame number and scorer of every goal. That header is the entire reason the project is tractable, because it means the cut list is generated rather than hunted for by scrubbing.
So there is a hand-written parser for it. Standard library only, no dependencies, because it has to run on a fresh Windows box with nothing installed but Python.
It had a self-test. Eight checks, building a synthetic header and reading it back. All green.
Then it had thirteen real files, which is better. All thirteen parsed. Every field came out looking like a plausible value of the right type.
I shipped it.
What a self-test actually proves
There is a line in the checking script I wrote later, and it is the whole post in one sentence:
A synthetic round-trip only proves it agrees with itself.
My test built a header using my understanding of the format and then read it back using my understanding of the format. If the understanding is wrong, the test is wrong in exactly the same direction, and it passes. A round trip through a single mental model cannot detect an error in the model. It can only detect an error in the code's fidelity to it, which is the easier problem and rarely the one you have.
The thirteen real files felt like they answered this, and they are a much better test, and they were also not enough. I will come back to why.
Building something to disagree with me
boxcars is a mature Rust parser for this format that other people maintain. I wrapped it in a tiny binary that dumps the header properties as JSON, and wrote a script that runs both readers over the same file and compares them field by field: every scalar, every goal's frame and scorer and team, the full player list.
Not as a replacement. The pipeline still uses my reader, because the no-dependencies rule is load-bearing on a Windows machine I am not sitting at. The Rust one exists purely as something to disagree with mine.
Thirty-three of fifty-three replays failed.
The first bug is a promise I ignored
The format stores properties as a name, a type, a size, and then the data. For a struct, the declared size covers the data only. There is also a name string sitting in front of it, and that name is not inside the size.
My reader was not using the size at all. It read the struct's fields one after another until it ran out of fields it recognised, and wherever it happened to stop, that was where it decided the struct ended.
That works right up until the struct contains something you have never seen. A newer build added a player-id field whose layout is platform-dependent and version-dependent, and my field reader walked straight off the end of it.
Here is the part that matters. A binary parser that loses its place does not crash. Everything after that point is still bytes, and bytes are still readable as lengths and integers and strings. The parser keeps going, confidently, reading structure out of the middle of other values, and hands you back a header full of numbers. A desynchronised parser does not produce an error. It produces fiction in the right shape.
The fix is not to learn the new field. That is the trap, and it is the version of this I would have written a year ago: decode the new thing, ship it, and be broken again on the next game patch.
The fix is to stop inferring where the struct ends and seek to where the size field already said it ends.
struct_name = r.text()
data_start = r.pos
try:
fields = read_property_list(r)
except ReplayFormatError:
fields = {"__unparsed": True}
end = data_start + size # authoritative, whatever happened above
Read what you can, then jump to the declared end regardless. If the fields inside were unreadable, say so in the output and carry on with the rest of the file intact.
A length prefix is the format telling you how to skip what you do not understand. A parser that ignores it has quietly signed up to understand everything in order to skip anything, which means every future addition by the studio is a fatal bug rather than an unread field. Using the size turns a whole class of future breakage into a shrug.
The second bug is one character wide
Strings in this format are length-prefixed, and the length is signed. A negative length means UTF-16, and the count is characters rather than bytes. A positive length means one byte per character.
I decoded the single-byte case as UTF-8. That is the reflex, and it is wrong. It is Windows-1252.
Almost every name is pure ASCII, where the two encodings are byte-for-byte identical, which is why thirteen files and a synthetic test all agreed with me. Then a player name with an accented character came through and turned into a replacement character, because that byte is not valid UTF-8.
The corrected version cannot fail at all:
return raw[:-1].decode("cp1252", errors="replace")
Every possible byte decodes to something under cp1252. There is no invalid input. The bug was not that I picked a rarer encoding; it was that I picked one with a failure mode for a field that had no way to report one.
The test set was the third bug
This is the part I keep thinking about.
The thirteen files were real. They were not synthetic, they were not mine, they came off an actual machine. They were also all from 2021 and 2022, because that is what happened to be on the laptop I collected from first.
The struct that broke everything was added to the format later. My corpus was a sample of history, drawn entirely from before the change. I was not testing my parser against the format; I was testing it against one era of the format, and the format is a moving target maintained by a company shipping patches on a schedule that has nothing to do with me.
A test corpus has a date range, and nobody ever writes it down. Thirteen files sounds like a sample size. "Thirteen files, all from an eighteen-month window that closed four years ago" sounds like what it actually was.
It reads now as 53 of 53 agreeing, across 2021 to 2026, and the span is the part of that sentence doing the work.
What this cost and what it bought
Two bugs. Neither one throws. Neither one is visible in any output a person would look at. One of them produced entire headers of well-formed nonsense and the other produced a single wrong character in a name.
And neither was findable without a second implementation to disagree with, because every other check I had was ultimately me asking myself whether I was right.
That is a slightly different failure from the check that scanned zero files and reported success. That one was looking at nothing. This one was looking at exactly the right things, running against real data, and still proving nothing, because both sides of the comparison came out of the same head.
The rule I am keeping: when you implement somebody else's format by hand, the self-test is for regressions and the oracle is for correctness, and they are not substitutes. Pull in the mature parser you were too proud to depend on, use it as a judge instead of a dependency, and let it tell you which twenty years of assumptions you brought with you.
It found two. I would not have found either.