The phantom storm, and four other bugs that rebuilt my monitoring

Ninety-one fixes, five worth retelling

Ninety-one of the first 184 commits were fixes. Most were ordinary. Five were not, and those five did more to shape this system than any feature I shipped. Each one is written up in the project's blameless lessons chapter in the same format: what broke, why, the fix, and the rule. I am quoting my own postmortems here, version tags included, because a failure story with the specifics filed off is just a parable.

1. The probe that always said ok

The health loop on the server had a check for the data pipeline. For weeks it reported ok. During some of those weeks the pipeline was clearly behind, the site was serving an advisory hours older than it should have been, and no incident ever opened.

The reason is embarrassing in the way the best bugs are. The probe concluded the pipeline was healthy because the loop running the probe was running. It never measured how old the data actually was. It measured itself, found itself alive, and reported that as the system's health.

The fix was to make the probe call the same freshness evaluator the public status page calls, so there is exactly one function in the codebase that can answer "is this data current" and both the outside view and the inside view are forced through it. They cannot disagree, because there is nothing left to disagree with.

v2.5.52, the rule: a monitor must measure the thing, not its own liveness. Freshness is computed one way, for everyone.

This is the failure I think about most, because a monitor that lies is worse than no monitor at all. With no monitor you check manually. With a lying monitor you stop checking.

2. The phantom storm

In late June the status page went to degraded and stayed there for four days. The cause was a storm called ep902026 that had been over for days and was still, as far as my system was concerned, an active tropical system in the East Pacific.

Storms in the ninety range are invests: areas the National Hurricane Center is watching that may or may not become anything. When the NHC stops tracking one, it does not delete the data file. The file sits in the directory for days afterward, unchanged. My retirement logic only deactivated storms that had disappeared from the feed entirely, so a file that lingered kept a dead storm alive forever, and the freshness probe watching that storm kept firing because its advisory content was ancient.

The fix was to stop inferring liveness and start reading it. The directory listing carries a last-modified timestamp for every file, so a storm still present in the feed but untouched beyond a configured window now gets retired on the upstream's own signal.

The sequel, which was worse

That fix produced a second bug, and the second one is the better lesson. Over the following weeks I got one 28-hour partial outage during Tropical Depression Cristina, plus a steady drip of short ones, every single time a storm dissipated. Two problems were compounding.

The first was a clock mismatch. Retirement fired on the file's modified time at 24 hours. The freshness probe that declares a service down fires on the advisory's content age at 16 hours. Content age always leads modified-file age, because of the lag between the NHC writing an advisory and posting it. So there was a guaranteed window, every time, where a storm was simultaneously active and down. The monitoring was not wrong. The two halves of the system were reading two different clocks.

The second was a flap. The ingest step hard-coded active: true on every upsert, so the sweep that retired a storm had its work undone on the next cycle five minutes later. Deactivate, reactivate, deactivate, reactivate.

Both were cured the same way: retire on the content clock, set the threshold below the probe's threshold rather than above it, and centralize the whole lifecycle decision in one module so ingest and the sweep cannot hold different opinions about whether a storm exists.

v2.5.54 and v2.5.60, the rules: use the upstream's own last-activity signal, not your derived state, to decide liveness. And retire a thing before its stale data can trip the probe that watches it: same clock, lower threshold, one decision point.

3. The five-minute cron that ran six times a day

The pipeline was scheduled by a GitHub Actions cron, every five minutes through hurricane season, executing on my own self-hosted runner. On paper that is clean: GitHub owns the schedule, my server owns the work.

In practice the trigger fired somewhere between six and eight times a day, with multi-hour gaps. GitHub's scheduled trigger is explicitly best-effort, and under load it delays and drops. The freshest advisory on the site routinely drifted past eight hours, which flagged the pipeline as degraded during completely normal operation and opened a recurring incident roughly every six hours.

I spent a while suspecting my own code, which was the wrong instinct and cost me a day. The actual root cause was a split I had not noticed I had made: execution lived on my machine but scheduling lived on someone else's, and the someone else had no obligation to be punctual. The schedule moved to a systemd timer on the server that does the work. The GitHub workflows still exist, but they are manual-only now.

v2.5.58, the rule: schedule recurring work where it runs. If something must happen on a real cadence, own the clock.

4. Every outage message was the same bland sentence

For a stretch, every incident, on the public status page and in the engineering ticket, said some version of "Some services are temporarily unavailable." I had written two careful, specific prompts to produce those messages and neither of them had reached a user in weeks.

Both prompts had max_tokens set to 512. The technical write-up ran longer than that. The response came back cut off mid-string, the parser threw Unterminated string in JSON, the code caught the error exactly as designed, and fell back to the generic template. Every time. The error handling was working perfectly, which is why nothing ever looked broken.

Three things changed. The token budgets got sized to the structured output rather than to a round number, 700 for the user-facing notice and 1500 for the engineering report. The prompts got deeper. And the fallback itself got rewritten so that even with the model completely unavailable, the message names the real impact for each affected service and says what still works.

v2.5.60, the rule: size limits to the structured output, not to a round number. And make the fallback path as good as you can without the dependency, because the fallback is what ships on the day the dependency fails.

That second half is the part that generalizes furthest. Everyone tests the happy path. The degraded path is the one your users actually meet.

5. The storm that lost its name

Tropical Storm Cristina appeared on the site as EP032026. So did post-tropical Amanda, as EP012026, after having been correctly named for days.

Names in this system come from the Tropical Weather Discussion, a text product issued every six hours. The tracking feed, which updates far more often, only carries the code. Every ingest cycle upserted the storm with the code as a fallback name, and both database adapters wrote name = EXCLUDED.name on conflict. So the real name was correct only during the window when the current discussion happened to list that storm in a section my parser could read. The rest of the time, the placeholder overwrote it.

The fix is a shared function both adapters call: a value that matches the fallback pattern never overwrites an existing real name, and any genuine incoming name always applies. The placeholder can only ever fill a gap, never replace something better.

v2.5.55, the rule: when a source re-supplies a placeholder every cycle, make the enriched value sticky on write. Never let a placeholder clobber real data.

There is a residual case I chose not to fix, and it is documented as a known limit rather than quietly patched: a storm named off the six-hourly cycle, in a special advisory, stays a code until the next discussion lists it in a form the parser handles. The sticky rule then preserves it. I would rather ship a written limit than an unwritten guess.

What these five have in common

Not one of them was a mistake in the feature. The tracker worked. The models were parsed correctly, the interpretation was good, the site was fast. Every one of these was a failure in the layer that is supposed to tell you the truth about the layer below it, and four of the five were invisible from inside the system while they were happening.

That is why this project spends far more of its design on monitoring than on features. The watchdog that reads what the operating system reports about each scheduled job follows one strict rule that came out of these five: no facts, no verdict. If the collector that gathers those reports is itself broken, the watchdog reports nothing rather than guessing. Eighteen checks watch this system and six of them exist only to watch the machinery doing the watching.

Every one of these fixes shipped with a failing test written first, and every resolved incident since has a postmortem attached, which is a rule in the project's constitution rather than a good intention. That record is also what made the next step possible. A system cannot be trusted to fix itself until you can trust what it says about itself, and getting there took these five failures and about two months.