Skip to content
PETROVA

2026-09-04-main-red-alert-webhook-wired

title: main going red now notifies, and cancellations are explicitly not failures date: 2026-09-04 status: closed

Section titled “title: main going red now notifies, and cancellations are explicitly not failures date: 2026-09-04 status: closed”

Context

Section titled “Context”

main reddened silently for a week on sync-docs (six consecutive runs, nothing downstream reading it — #393 fixed the cause) and again for four days on the CI-billing outage (F-130/F-140). Both were found by a human opening the repo, not by any instrument. This wires the first notification for exactly that class: main goes red, someone finds out without opening GitHub.

A fresh, dedicated webhook (WEBHOOK_PETROVA, GitHub Actions secret) — not mary-wiki’s discord-wake/discord-brief webhooks. Those are class-A and class-C wake transports for a different estate, held box-only per their own alertmanager.yml (“never in git and never in a pull request”). This is a different, lower trust class: high-volume, low-value, trivially revocable. Its worst-case leak is a stranger spamming a channel nobody acts on — the box-only credential never moves, and a second one is created at a lower class instead. That reclassification is the whole justification for putting it in GitHub Actions secrets at all.

Decision

Section titled “Decision”

.github/workflows/main-red-alert.yml, triggered by workflow_run on the ten workflows that actually run on: push: branches: [main], posting to Discord only if: conclusion == 'failure'.

Cancellations are explicitly excluded, and this is not incidental. gh pr checks reported a concurrency-group cancellation as fail earlier in this same session — Registry slug scope on #394 showed fail, was actually cancelled, and only came back success on manual re-dispatch. A !success() or a gh pr checks-grep condition would have paged on that exact case, and every routine force-push cancellation after it. workflow_run’s own conclusion field distinguishes cancelled/failure/success natively, so this workflow fires only on the one that means something broke.

Scoped to main only. PR failures are already visible in the PR; main going red silently is the class this exists for. Alerting on both is how a channel earns muting inside its first ten messages.

Payload: repo, workflow name, commit SHA, run URL. Nothing else.

Rejected

Section titled “Rejected”

Alertmanager inbound (9093/tcp published on ODIN) — considered and killed outright, not deferred. Alertmanager’s API has no host binding at all today (9093/tcp: null), which is the correct posture, not a gap. Publishing it to accept POSTs from GitHub-hosted runners means accepting them from the internet — no stable IP range to allowlist — which makes /api/v2/alerts forgeable by anyone who finds it, on the same pipeline that carries mary-wiki’s class-A wakes. Strictly worse than the silent gap it would close. Not reconsidered here; if it comes up again it needs its own decision from scratch, not a citation of this one.

Verification

Section titled “Verification”

Run 2026-09-04, via #397 (merged, then reverted). citation-resolution.yml edited to exit 1 unconditionally, landed on main at c78d67b7. Citation resolution run 33880580702 — failure, as forced. Main red alert fired three times on that push (one per watched workflow that completed): success for the failed Citation resolution run, skipped for the two others that completed success on the same push — the if: conclusion == 'failure' guard held exactly as designed, job-level skip rather than a broken trigger.

Message received in #petrova-ci, confirmed against the actual run, byte for byte: workflow Citation resolution, commit c78d67b770893b01c6abc43ac8eebfd73b2cee5f, https://github.com/petrova-codes/petrova/actions/runs/33880580702 — the correct run id, not the alert workflow’s own. The break was reverted the same session (this commit).

Amendment 2026-09-04: --fail-with-body added and falsified

Section titled “Amendment 2026-09-04: --fail-with-body added and falsified”

Bare curl exits 0 on a Discord 404 — a revoked or deleted webhook would have left the notify job reporting success while posting into the void, reproducing this instrument’s own defect class inside itself. Fixed (#399, --fail-with-body) and falsified before trusting it: WEBHOOK_PETROVA temporarily set to a deliberately invalid URL, Citation resolution forced to fail again (#400), Main red alert’s notify job observed completed failure against the bad webhook — confirmed red, not silently green. Secret restored, break reverted (#401).

Sign-off

Section titled “Sign-off”