How HurricaneWise ships its own repairs

The default rule and the exception

On my other systems an AI digs into a failure and prepares a fix, and the fix sits there until somebody reviews it and lets it into production. HurricaneWise ran like that for most of its life. It detected its own incidents and wrote up the report with the evidence linked, and then it waited for me.

This summer I changed that for repairs. When its monitoring detects a failure, HurricaneWise now tries to fix it and ships the fix on its own. I still handle anything that would change what the product is, like a new feature or a layout change.

The steps

1. Monitoring notices. There are nineteen health checks. Four of them run from outside the infrastructure as Grafana synthetic checks (the fastest every 60 seconds, the others every 3 minutes), and the rest run on the server's 15-minute loop or on the status route. Six of the nineteen watch the monitoring itself and only go on what the OS reports. An incident needs two bad checks in a row, since a single blip is usually just the internet. Then the system opens an issue against itself saying what's down and what still works.

2. It dispatches itself. Any confirmed incident qualifies. There's no severity threshold and no approved list. The only exception is an incident that affects the resolver itself, and those go to a person.

3. A sweep picks it up. The sweep runs every five minutes and starts an AI session with nobody at the keyboard, capped on money and time. It holds only its dedicated ops API key, with no GitHub, database, storage, vault or deploy credential. A lock makes sure only one session runs at a time.

4. It writes the fix and proves it. The orchestrator around the session opens the pull request, because the session doesn't have a GitHub credential.

5. CI has to pass. That's lint, types, the full test suite and the browser tests on desktop and mobile. If anything is red there's no merge, and the orchestrator will only merge the pull request this session produced.

6. The merge tags a release and the tag deploys. After that the system checks the live site's health before it calls the fix done.

7. It writes the postmortem and emails me the trail, whether the fix worked or not.

8. Any failure escalates. If something goes wrong at any stage, the incident gets marked as needing a person and nothing is deployed. It goes up on the ops dashboard (behind the zero-trust check) and I get an email.

What I wrote down before the first session

Fixes come in as pull requests. The session never pushes anything. The orchestrator pushes fix branches only and merges only the pull request the session produced, and that's enforced in code.

Merging is up to CI. The fix goes through the same checks as a pull request of mine. What the session says about its own work doesn't count toward the merge.

The deploy gets checked on the live site. I've shipped a green deploy onto a broken page before, so after the tag there's a health check against the real site.

Credentials. The session runs in a workspace that's reset to a clean copy of main before and after every run, so whatever one session leaves behind isn't the next one's starting point. It has no GitHub credential and no database, storage, vault or deploy credential either. It can't get to production data.

Every fix has to be traceable. Incident, issue, pull request, checks, merge and postmortem, each linked to the next. The constitution requires that chain, and a test checks for it.

Spending limits. There's a $15 soft cap per session that flags it for review and a $50 hard ceiling per day. A session is also limited to 80 turns and two hours, there are at most 12 sessions a day, and repairs run on their own API key so that spend doesn't mix with the product's.

Resolver incidents go to a person. When the resolver breaks, that opens an incident too. A session sent to fix it would probably fail on the broken resolver and open another incident, and so on. So a guard skips dispatch for any incident touching the resolver's own services. Those come to me.

Two switches. The internal dashboard sits behind a zero-trust check (Cloudflare Access) that fails closed. One switch there turns the resolver off and the other turns off its permission to deploy. I haven't had to use either one yet.

What it may change

The permission only covers repairing what the product already does.

That includes the product as it is now, the data it gathers and stores, the incident pipeline and the watchdog over the monitoring. In practice it's usually an upstream source that changed its format, or a job that stopped running.

New features, new data sources, new pipelines, layout and interface changes and performance work all come to me. Feedback sent through the site goes into the same tracker as everything else, and the system doesn't act on it on its own. I'd like it to be fully autonomous eventually, but product judgment is still mine, so every feature request and every piece of user feedback waits for me.

I kept the product frozen for this. I was testing whether the running could be handed over, and if features were changing at the same time, a new failure could come from either one.

That boundary is held by the instructions the session runs under and by the shape of the incidents it gets. There's no technical restriction on which files it can edit, so in principle a session could redesign a page and the code wouldn't stop it. The session is told to make the smallest fix, and the test suite has to stay green. Neither of those would catch a change that passes the tests and goes outside the job. I'll tighten this as I see what sessions actually touch.

Tuning the limits

The settings I kept changing were turns per session, wall-clock time and the dollar amount where a run gets my attention. With a low turn limit, sessions ran out two steps from done and the money was already spent. When I raised it and gave the clock more room, a session could drift into a file that had nothing to do with the failure, and nothing in the code stops that.

Right now it's 80 turns, two hours, $15 a session before I look and $50 a day before everything stops. I got there by watching runs and I expect to change them again.

The same kind of tuning happens earlier, in deciding what counts as a failure. It's two bad checks in a row. The outside checks run every 60 seconds to 3 minutes for what a visitor would notice, and the 15-minute internal loop covers slower things. If I set that too tight I get false incidents, and every one of them now starts a session that spends real money investigating nothing.

Longview is set up differently

I also run Longview, which manages a household's records and its long-term funding plan, and it's private by design. Its repair agent is structurally barred from merging or deploying anything, and its spec requires a test in that build proving it can't reach the credentials that would let it.

I decide that line for each system. Longview holds a family's financial plan, so a mistake there reaches a lot further. On HurricaneWise a bad automated fix means a stale weather page and an email to me. The page regenerates all the time and an answer that fails its checks publishes nothing. A deploy also rolls back in minutes.

Fixing the monitoring first

A few months before the resolver, the monitoring was still reporting failures that weren't real.

So the resolver waited until I could trust the monitoring. One probe was reporting its own liveness as the health of the whole system. A storm that had already dissipated in early June 2026 kept opening false outages for days. And incident messages were collapsing into one generic sentence because a token limit was cutting off the JSON. Fixing all that took about two months.

Cost

The resolver is the reason the AI line went from about $3.60 a month to about $7. The server went up too, from $5.25 to $7.60, so by my cost model's estimate the total went from about $9 to about $15 a month. The extra AI spend is repair work I used to do myself in the evenings. It's capped and metered on its own key, and I see it in the same cost snapshot I run for everything else.

The permission is written into the constitution, which is versioned, and every use of it sends me an email.