Well formed answers that happened to be false

Caught by a second, independent route

The most common entry on this page, by a wide margin. Not a better test of the same code, and not a conservation law: a different implementation, or a different measurement, arriving at the same quantity from somewhere else.

Three Times the Physics Said No

There were two Earths in the program and one of them was wrong.

Caught by A planet table carrying both a semi-major axis and an independently observed mean motion, which Kepler’s third law says are not independent. "The only thing that ever revealed it was building a second, independent Earth and asking whether the two agreed."

A Second Opinion, and It Lied Too

A geometry pipeline that is easy to get subtly wrong and still returns a well formed matrix.

Caught by Two independent runs agreeing to four digits, which is the trap rather than the check. Agreement between two things that share an assumption is not evidence about the assumption.

Right for the Wrong Reason

A well formed answer produced by arithmetic rather than by the world.

Caught by A second route to the same number, and nothing else.

I Said It Works in Water. It Does Not.

A drag coefficient of 0.000000035 for a perfectly sane thirteen percent section. Every guard passed: thick enough, enough area, surfaces uncrossed, not separated. Skin friction alone on a flat plate of that length is 0.003.

Caught by Comparing the two routes to the profile drag that the file already computed and had never compared. They agree between 0.705 and 1.170 across 69 healthy sections; this sat at 0.0000089. Four further rounds of guards each produced a shape parked on the new edge, which is the finding rather than the fix.

What the Optimiser Was Not Told

A drag coefficient computed with the wrong trailing-edge closure: 0.004101 where the module said 0.004181. Two percent, comfortably inside what anyone reads as rounding.

Caught by The cross-check at the foot of the figure script, which re-prices both sections and requires the module’s published pair back. It was the only thing on screen that knew.

The Only One That Said Yes

An exhaust velocity that could have been transcribed instead of derived.

Caught by Deriving it from three named losses and landing within three percent of the figure the literature quotes, which makes the quoted number a check rather than an input.

The Part With No Shape

Two numbers that looked like two routes to one quantity, where the small one appeared to catch the large one out.

Caught by Asking what each number was actually a number of. They were different questions sharing a word, so the comparison was never available.

Where the check that looked sufficient was not

The sharpest corollary in the archive, and the one that cost the most time: some checks feel like independent confirmation and are structurally incapable of catching the error.

The Model That Cannot Make a Puddle

A flood solver whose outflow limiter was wrong in three successive ways. Two of the wrong versions oscillated while conserving mass to eleven figures. A later bug flooded a draining plane to 285 mm instead of 4 mm with the mass balance exact throughout, because the water was conserved into the wrong place.

Caught by Comparing against a bisection on the geometry. "A conserved quantity is not a second route." The wrong orbit can be conserved perfectly.

Two People Saw a Light

A two-station trajectory solve has no residual at all. Two planes always intersect in a line, so however hopeless the geometry, the arithmetic still returns a trajectory and nothing is left over to flag it.

Caught by The convergence angle, which is a fact about where the observers stood rather than about what they saw. Below five degrees the solver refuses. You can walk it into that state on the page.

A Working Compass Pointing the Wrong Way

A sign flipped in the north component sent every azimuth through east-west. Elevation was right, the roll invariance was right, and turning on the spot still swung the aim correctly.

Caught by A test of the one thing the other tests did not cover. Everything that was checked passed, which is what a working compass pointing the wrong way looks like.

Caught by building something that can say "I do not know"

Several of these were fixed not by making the answer better but by giving the system the ability to decline, and then finding out how often it declined.

One Camera Sees a Plane

Three separate cases, each returning a well formed answer that happened to be false.

Caught by The same fix all three times: "decide what ‘I do not know’ looks like, and make the system capable of saying it."

I Built the Check That Refused My Own Drive

A hyperbolic tangent saturating to exactly one in double precision, so the relativistic rocket equation returned exactly the speed of light. "Printed in a table it would be a perfectly well formed claim to have reached light speed, produced by rounding."

Caught by Carrying a saturation flag alongside the number instead of trusting the number.

You Cannot Steer It From Inside

A confident, well formed number describing a bubble whose crew cannot signal its own front.

Caught by A geodesic integrator built specifically to check the model against itself by a second route, and a scan rather than a single ray: the condition is refuted by one geodesic and confirmed by none.

It Never Said Where

A tool that reported the size of a search area and never once printed a position. "Four posts arguing that a well formed number proves nothing, and the tool had been withholding the only one that mattered."

Caught by Using it for the thing it was built for.

Caught by looking at the thing, not at the code

A picture, a page or a machine that is wrong looks exactly like one that is right. None of these was findable by reading the source.

A Sky in Front of the Stars

A star field that rendered correctly and hid most of what was behind it. "A well formed image, plausible in every detail, produced by something other than what it appeared to be."

Caught by Counting what was actually on screen rather than trusting that it drew.

Standing Where the Observer Stood

A layout that "looked fine on the screen it was built on."

Caught by Measuring the rendered boxes on a device that was not that screen.

Zero Issues Found

A system returning a well formed answer to the wrong question, where "there is no amount of staring at the screen that fixes it, because the screen looks correct."

Caught by Checking the question rather than the answer.

The Saw Came With the Wrong Blade

A well built machine, everything in the box as listed, and the blade on top says woodworking. Nothing broken and nothing missing.

Caught by Reading the blade before putting a stone through it. "A well formed kit is not a kit for your job."

Caught by measuring instead of asserting

A number that is stated rather than computed cannot be wrong in a way anything notices, which is the problem with it.

Fifty-Four Trillion Miles

A ledger capped at five thousand rows "returned a well formed total that was short by whatever fell off the end, and the total was believed."

Caught by Asking the ledger how many rows it had, rather than reading its total.

The Most Dangerous Number

A dashboard that "looked fine. It always looked fine."

Caught by Checking it against something that was not the dashboard.

Imported by Nothing

A colour reference applied across frames, where "applying one frame’s cast to another frame’s mean is a well formed wrong answer."

Caught by Two independent checks that catch different parts of the range, one from the file’s own account of itself and one from the pixels.

Opening the Barrel Early

The same failure outside software: "the process completes, the output is well formed, and it is wrong for a reason that was invisible at the time it was introduced."

Caught by Opening the barrel early and looking, instead of running the stage the guide prescribes.

What this adds up to

Every engine on this site is built to the same three rules, and all three came out of the entries above rather than being decided in advance.

Two independent routes to every number that matters. The physics modules are plain dependency-free JavaScript; the figures are drawn by scripts that solve the same problem again in numpy from the equations rather than from the module; several of the interactive pieces solve it a third time in your browser. Each script prints its own figures beside the module's when it runs.

A check that never fires is worse than no check. It sits in the output looking like diligence. One validity criterion in the flood engine was replaced for exactly this reason: it got more confident as the ground got flatter, which is the opposite of the warning it was meant to give.

Refusing is a feature and it ships. Seven interactive pieces on this site can be driven into a state where they decline to answer, and in most of them that state is the interesting one.