I have spent a few days on a toolchain that turns gameplay captures into short vertical edits. Most of it is Python, a fair amount of it is PowerShell because the recording happens on Windows, and as of this week it works.
I think.
That last part is the subject of this post. I want to be precise about what is carrying the "I think," because for a while the answer was something that should have been carrying the opposite.
The check
Windows still ships PowerShell 5.1 by default. PowerShell 7 exists, it is better, and it is a separate install that most machines do not have.
This matters more than a version number usually does, because a PowerShell 7 operator is not a runtime warning on 5.1. 5.1 fails to parse the file. The script does not run and hit an error partway through; it dies on load, having printed nothing at all. For a script somebody runs once, in a hurry, on the one evening they set aside for it, that is close to the worst failure mode available.
So there is a check for it. It walks every .ps1 in the tools directory, parses each one, and flags anything the older runtime cannot read. It matches on token kind rather than searching the text, so a ?. inside a comment or a string does not trip it.
It earned itself immediately. The recording script shipped with a null-conditional ?. in it and would have been unusable on a default Windows install. The check found it before it cost a real evening. Good check.
What it was doing on Windows
The scan path was hardcoded:
Get-ChildItem /home/user/<project>/tools/*.ps1
That is the Linux box I write on. It is not a path that exists on Windows.
On Windows, that glob matched nothing. The loop ran zero times. No file was parsed, no token was examined, and at the end the script printed its summary in green:
0 incompatibility issues
Which was true. It had found zero. It had also looked at zero.
The only machine the check was wrong on was the only machine it existed to protect. On Linux, where the path is correct and where PowerShell 5.1 is not even the question, it worked perfectly and told me so every time I ran it. The failure was invisible precisely where it mattered, and the success was reliable precisely where it did not.
Why this is worse than not having it
Not having a check leaves you where you were. You know the risk is unhandled, so you are careful about it in other ways: you read the diff, you test on the real machine first, you stay suspicious.
A check that passes spends that caution. It is an asset you draw against. Every green run buys a little more confidence to spend somewhere else, and a check that passes by finding nothing sells you that confidence without holding anything behind it.
This is the same shape as the ledger bug I wrote about this morning, where a display limit was quietly deciding which lots a sale consumed. In that case the output was a number that was wrong. Here the output is a number that is right, technically, about a question nobody asked. Both render identically to the correct thing. Neither errors.
I have come to think this is the whole genre. A system that breaks tells you. A system that returns a well formed answer to the wrong question does not, and there is no amount of staring at the screen that fixes it, because the screen looks correct.
The missing thing is a denominator
The report said what it found. It did not say what it looked at.
Every summary line of this shape has the same hole in it. "No errors." "All clear." "0 issues." Those are numerators. Without the denominator beside them, they are all satisfied by an empty set, and an empty set is what every broken check produces.
The fix is two lines, and the smaller one is the one that matters:
$scanned = 0
foreach ($f in Get-ChildItem (Join-Path $PSScriptRoot '*.ps1')) {
$scanned++
...
}
if ($scanned -eq 0) { <fail loudly> }
$PSScriptRoot instead of the literal path fixes today's bug. The counter fixes the class of it. If that glob ever matches nothing again, for any reason at all, the check fails rather than congratulating me.
There is a comment above it in the source now, because the next person to touch that line will not have lived through this:
A check that passes by finding nothing is worse than no check, because it is believed.
Negative controls, everywhere
Once you have been caught by a vacuous pass, you start seeing candidates for it everywhere, and I went through the rest of them.
Every check in that repo now has a case that must fail, on the principle that a test which only ever passes proves nothing about itself:
- The path discovery asserts it returns null when the game is genuinely absent, so "found it" is distinguishable from "always says yes."
- The encoder probe asserts a working encoder still returns success, so "failed" is distinguishable from "always says no."
- The audio search, which matches loosely on purpose because a downloaded file is almost never named what the song is named, asserts that "Talk That Talk" does not match a search for "Don't Talk." Loose matching that matches everything is not loose, it is broken.
- The isolation routine, which temporarily moves files out of a folder, asserts that it is not secretly a no-op. It would otherwise pass by doing nothing, which is exactly the failure it needs to rule out.
One more that I liked. A self-test has to leave nothing behind, and one of them did not: its undo ledger was a module-level constant pointing at the real folder, so a test run against a temporary directory still wrote into the repository. There is now a check asserting the repo's own folder is untouched after the tests run. A test suite with side effects is measuring a system it is also modifying.
The part I cannot check
Here is where the "I think" actually belongs, now that it has been moved off the checks and onto something real.
Every suite in that project passes, and I now believe the passes mean something, which I did not a week ago. But the single most important path in the whole pipeline is desktop capture on real Windows with a real GPU, and there is no machine here that can run it. The code probes for hardware encoding and falls back when the probe fails, and that fallback has never been exercised against an actual driver.
So the honest statement is not "it works." It is: everything that can be checked from here is checked, the checks have been checked, and the first real recording is still the first real test of the part that matters most.
That is a much smaller claim than the one I would have made on Tuesday, when a green line of text was reporting no problems across fourteen files and the truth was that it had not opened any of them.