I assumed the models would be the hard part. They weren’t.
For the last few months I’ve been building a suite of inference services that score data inline as it arrives: models trained from data the system already collected, scoring on the way in, the verdict landing as ordinary fields next to the measurement. Not an API you call beside the pipeline — inside it. On CPU, milliseconds per document, which is what makes the arrangement thinkable at all.
This is the part of lean IoT infrastructure I’ve been stuck on lately, and it follows the others I’ve written up this year: Keystone for the edge runtime, OTA updates over NB-IoT for keeping a fleet alive over a link that drops, and the economics of the platforms you’d otherwise buy. The reason this one is worth writing down separately is that it is where I was most wrong about which part would be hard.
Every failure worth writing about lived in the same place: the seam, where the pipeline meets the models. Every one was silent. Not one was caught by a unit test.
Four are worth writing down, because I don’t think they’re specific to my implementation.
1. Green transport does not mean it works#
I have a conformance suite that points the production client at a live inference process and asks, per model family: can this take part in the data path? Is the training route served, does it read the file from the field we send, what shape does job status come back in, is there a batch verb.
One family scored 4 out of 4 green and could not process a single real document.
The mechanism generalises. When a domain is prepared, the system decides which columns are evidence and drops the rest: a timestamp that is unique per row is an identifier, not a signal — keep it and the model memorises rows. A device id says which device this is, not something about the device. Both decisions correct, both reported in the training log.
But the input contract it generated then declared additionalProperties: false over the
surviving columns. So every real document was rejected — because a real document carries
its timestamp and its device id. Dropping a column and then refusing payloads that
contain it is a contradiction, and it surfaces at the exact moment the model looks
healthiest: it trained, the parity check passed, the bundle verified.
And ingest scoring is fail-open per document — deliberately, because a slow or missing model must never take an ingestion down with it. So the observable symptom was: zero verdicts, zero errors, zero log lines.
The whole failure fits in one picture. Everything on the training side is correct, and that is the point:
That same shape has caught me four times in different places. It isn’t a bug, it’s a class: a contract that is correct in isolation and contradictory in composition.
2. The bottleneck was not the model#
Putting scoring into the ingest path dropped throughput from 11,200 to 6,900 messages per second. The obvious diagnosis — inference is slow, batch harder — was wrong.
What changed was the document. The verdict makes each one 8× larger for anomaly detection and 5.6–9.2× for classification, growing with the number of classes because the probability map carries one entry per class.
You are not paying for inference. You are paying to serialise, write and store a document eight times bigger, permanently. That is not a performance problem hardware fixes; it is a decision about how much of a verdict is worth keeping — a far more useful thing to learn. And you only learn it with the model in the path: an API returning predictions would never have shown it, because the document would have stayed the same size.
3. Consolidation created shared fate#
Five inference processes became one, so that a consumer configures one address. The code documented that a corrupt bundle in one family leaves the others serving. It did the opposite: any load failure aborted startup, so one unreadable bundle out of eleven stopped everything.
For a while, one process was strictly worse than five. Nobody noticed, because no test had ever started all five families together.
The same blind spot hid a second problem underneath. Each family carried its own copy of the same bootstrap, each wrapped in its own run-once latch, sitting on a runtime that only initialises once per process. Across five processes that was invisible: every copy ran first in its own address space. In one process, three of the five families died the first time they had to share it.
That one is worth sitting with. Five copies of a bootstrap didn’t merely cost reading time — they made the target architecture impossible, invisibly, until the day somebody tried it.
4. One idea, three shapes#
Job status came back three different ways depending on which family you asked: flat, nested under a key, or flat without the log.
A client reading the status field at the top level finds nothing in the nested variant. It doesn’t error — there is no field there to disagree with. It waits, politely, forever, for training that finished twenty minutes earlier. And because the caller looks stuck rather than broken, the first instinct is to go and inspect the training job, which is fine, and finished, and innocent.
Five copies of a control plane, evolving separately, produce this reliably. Nobody designs three shapes for one idea; three shapes are what you get when the same idea is implemented five times and only three of them get reviewed on the same day.
The pattern#
None of the four was a model failure. None was visible in a unit test. And three of them were silent because of correct decisions elsewhere: fail-open scoring, per-document isolation, honest column dropping. Each of those is the right call on its own. Together they build a system that fails without saying anything.
Which is why the seam is where the work actually is. And none of it disappears if you move the models beside the pipeline instead of inside it — it just stops being anybody’s job, and waits for whoever ends up integrating the two.
Where the human decision still lives#
Putting models in the path does not mean the thing configures itself, and a system that pretends otherwise has started lying to its users.
Clustering has to know how many groups to split the data into. An expert knows; most users don’t and shouldn’t have to. So when nobody says, measure it — a bounded sweep scoring each candidate, with every score written into the training log. A value the system picks on your behalf has to be arguable afterwards.
Then the detail I nearly missed. Retraining re-runs preparation. If that number were
measured again on every pass, it would drift as data arrives — and the cluster labels
would drift with it. Today’s cluster_1 would stop being the one somebody wrote a rule
against. Silently, with nothing in the API saying so.
So: measured once, then pinned. And if a user supplies their own value, theirs wins, on every pass.
That generalises far past clustering. A derived value becomes a contract the moment somebody builds on top of it. Deriving it is the easy half; knowing when to stop deriving is the half that decides whether your automation helps or bites in three months.
One limit worth stating plainly, too: automatic feature engineering does not exist. What exists is honest preprocessing that says what it dropped and why. When a problem genuinely needs its own code, the answer is not to pretend the system divines it — it’s an explicit door to Python, with a clear contract about which artifact gets deployed.
What I’d tell myself before starting#
Test the composition, not the parts. Every one of these four passed its own tests and its own review. What none of them had was a test that started everything together, pointed the real client at it, and pushed a real document through — the only thing that would have caught any of them.
And if there’s one line to take from this: a contract that is correct in isolation can be contradictory in composition, and nothing in your test suite is looking for that.
Hero photo by Peter Herrmann on Unsplash.