The phantom storm and four other bugs

Five of the 91 fixes

91 of the first 184 commits were fixes. Five of those changed how the system is monitored, so those are the ones I'm writing up. Each has a write-up in the project's blameless lessons chapter and I quote those postmortems below, version tags included.

1. A probe that never checked the data

The health loop on the server had a check for the data pipeline, and for weeks it reported ok. During some of those weeks the pipeline was clearly behind. The site was serving an advisory hours older than it should have been, and no incident opened, because the probe only checked that its own loop was running and never read how old the pipeline data was.

The fix was to have the probe call the same freshness evaluator the public status page uses. There's exactly one function in the codebase now that answers "is this data current", and both the status page and the server probe call it.

That shipped in v2.5.52.

A probe that checks only its own liveness can stay green while the advisory ages. That is why this probe now measures freshness using the same evaluator as the status page.

2. The phantom storm

In early June 2026 the status page went to degraded and stayed there for days. The cause was ep902026, a storm that had been over for days but that my system still thought was an active tropical system in the East Pacific.

Storms numbered in the nineties are invests, areas the National Hurricane Center is watching that may or may not turn into anything. When the NHC stops tracking one it doesn't delete the data file. The file just sits in the directory, unchanged, for days. My retirement logic only deactivated storms that had disappeared from the feed, so a leftover file kept a dead storm alive indefinitely, and the freshness probe for that storm kept firing because its advisory was ancient.

The directory listing has a last-modified timestamp for every file. So now a storm that's still in the feed but hasn't been touched within a configured window gets retired on the upstream's own signal.

What that fix broke

That fix caused a second bug. Over the next few weeks I got a 28-hour partial outage during Tropical Depression Cristina and a steady run of short ones, one every time a storm dissipated.

Part of it was a clock mismatch. Retirement fired on the file's modified time at 24 hours, but the freshness probe that declares a service down fires on the advisory's content age at 16 hours. Content age always runs ahead of file age because the NHC writes an advisory before it posts it, so every time there was a window where a storm was active and down at once. And the probe was right every time.

On top of that there was a flap. The ingest step hard-coded active: true on every upsert, so a storm the sweep had retired came back as active on the next cycle (five minutes later, in that version).

Retirement now runs on the content clock with a threshold below the probe's, and the lifecycle decision lives in one module, so ingest and the sweep agree on whether a storm exists.

Those went out in v2.5.54 and v2.5.60.

3. A five-minute schedule that fired six times a day

The pipeline ran on a GitHub Actions cron every five minutes through hurricane season, on my own self-hosted runner. So GitHub kept the schedule and my server did the work.

In practice it fired six to eight times a day, with gaps of hours. GitHub's scheduled trigger is best-effort and under load it delays and drops runs. The newest advisory on the site kept drifting past eight hours old, which marked the pipeline degraded during normal operation and opened an incident roughly every six hours.

I spent a day looking for the bug in my own code first. The work ran on my machine but GitHub owned the schedule, and GitHub doesn't promise to run on time. I moved the schedule to a systemd timer on the server that does the work. The GitHub workflows are still there, manual-only.

That was v2.5.58.

4. The same outage message every time

For a while every incident said some version of "Some services are temporarily unavailable," on the status page and in the engineering ticket. I'd written two careful prompts to produce those messages and neither one had reached a user in weeks.

Both prompts had max_tokens set to 512, and the technical write-up ran longer than that. The response came back cut off mid-string, the parser threw Unterminated string in JSON, and the code caught the error like it was designed to and fell back to the generic template, every time. The page was still showing a notice, so nothing looked broken.

I made three changes. Token budgets got sized to the structured output, 700 for the user notice and 1,500 for the engineering report. The prompts got more detailed. And I rewrote the fallback so that even with the model down, the message names the real impact on each affected service and says what still works. Since 31 August 2026 a text field that runs over its length limit gets cut to length instead of failing the whole answer.

That was v2.5.60. Before it, the fallback was the message that ran every time the parse failed.

5. Storm names turning back into codes

Tropical Storm Cristina showed up on the site as EP032026. Post-tropical Amanda did too, as EP012026, after days of having the right name.

Names come from the Tropical Weather Discussion, a text product issued every six hours. The tracking feed updates a lot more often and only carries the code. Every ingest cycle upserted the storm with the code as a fallback name, and both database adapters wrote name = EXCLUDED.name on conflict. So the real name was only right while the current discussion listed that storm in a section my parser could read, and the rest of the time the placeholder overwrote it.

The fix is a shared function both adapters call. A value that matches the fallback pattern never overwrites an existing real name, and a real incoming name always applies, so the placeholder only fills a gap.

That shipped in v2.5.55.

There's one case I decided not to fix, and it's written down as a known limit. If a storm gets named off the six-hour cycle, in a special advisory, it stays a code until the next discussion lists it in a form the parser handles. After that the name sticks.

What I changed after

All five bugs were in the monitoring and the plumbing around it. The models parsed and the site stayed fast the whole time.

After these I spent more design time on monitoring than on features. The watchdog that reads what the operating system reports about each scheduled job came out of them. If the collector gathering those reports is broken, the watchdog reports nothing. And when the external checks and the server's telemetry disagree, the external checks win.

Each of these fixes started from a failing test, and every resolved incident since has a postmortem. I didn't let the resolver act on incidents until the monitoring had been through these five, which took about two months. It works from that record now. A false incident would start a paid repair session.