The four failures we have written about so far were all silent. This one was not. The system reported the problem on every single affected run, in plain English, and we read it every time.
It reported it as a warning. Six checks had been quietly not running for months.
How it looked
Two entries appeared at the bottom of the findings list, below the things that mattered:
That is the system being honest about its own limits — the behaviour we designed for deliberately, because "not measured" must never read as "fine". It worked. It told us. And because it told us in the same register as "damper free area exceeds the shaft cross-section", it read as housekeeping.
We had built something that reports its own blindness accurately, and then failed to notice it doing so for months. Reporting a problem and communicating one are not the same act.
What was actually wrong
The check register needs two things: the run output, and a record of what the model declared — the design fire, the extract duties, the leakage schedule, the device names. Without that second half, checks that compare achieved against declared have nothing to compare against, so they report themselves unevaluated rather than guessing.
That record is written when the deck is generated and stored with the run. The component that runs the register looks for it, and had one shortcut: if a caller passed a record in directly, use that instead.
Two report jobs did pass one in. As a convenience, they supplied the two values they happened to have to hand:
So every register run behind a report executed with no design fire, no leakage, no ventilation duties and no reconciliation. Six checks degraded to unevaluated, and the two warnings above dutifully said so.
Why nobody chased it
Because the warning was true, and truth is not the same as urgency. "Some checks could not be evaluated" is a sentence that invites you to assume the missing ones were minor. Nothing in the finding said which checks, or that the cause was structural rather than a property of that particular model.
It was also self-consistent. Every report said it, so nothing stood out. A warning that appears on one report in fifty gets investigated. A warning that appears on all fifty becomes the wallpaper.
What it was hiding
We found it because a reconciliation check — new, and looking at something else entirely — reported that a deck had never been checked against its model, on a run whose stored record showed all eight declarations conforming. Two statements that could not both be true.
Fixing the record source did not just restore the missing checks. One of them immediately failed:
The fix itself is dull: the stored record is now the base, and caller-supplied values are layered on top rather than replacing it. Merging is what the shortcut always meant. It took four lines.
The part worth taking away
Every serious analysis tool has a category for could not evaluate. Having one is good practice — the alternative is a tool that silently treats absence as success, which is worse in every direction.
But the category creates a second problem that nobody designs for: an unevaluated check is indistinguishable from a passed one at a glance, and both of them are quiet. The finding is filed at warning severity because individually that is proportionate. In aggregate, "six of thirty-one checks did not run" is not a warning. It is a statement that the register did not happen.
Not-evaluated is a finding about the assessment, not about the building. Those two things should not appear in the same list at the same weight.
What we changed beyond the bug: the count of unevaluated checks is now something to look at on its own, not a line among findings. If a register cannot execute a fifth of itself, that is the headline, whatever the model did.
If you commission CFD
This one is easy to ask about, and almost nobody does:
- How many checks ran, out of how many exist? If the report cannot answer, the register has no coverage statement, and you cannot tell a clean report from a mostly empty one.
- Which checks did not evaluate, and why? "Insufficient data" is a reason. "Not applicable to this building" is a different reason. Both are fine; not knowing which is not.
- Has that list changed between revisions? A check that quietly stopped running between issue 1 and issue 3 is exactly the kind of thing nobody looks for.
Coverage is reported before findings are
Every report states how much of the register actually executed. An unevaluated check is named, with its reason, and never counted as a pass.
Arrange a demonstration →