Skip to content
PETROVA

2026-09-04-scheduled-workflow-digest-wired

title: scheduled-workflow failures and a stale deploy-batch flush now get a once-daily digest date: 2026-09-04 status: closed

Section titled “title: scheduled-workflow failures and a stale deploy-batch flush now get a once-daily digest date: 2026-09-04 status: closed”

Context

Section titled “Context”

main-red-alert.yml watches workflow_run for the four push-triggered workflows it names, and cannot see a schedule-triggered failure at all — nothing pushes to main when a cron fires. This is the follow-up route named in docs/decisions/2026-08-31-host-production-deploys-move-to-the-hook.md’s “Re-verified 2026-09-04” section: “no deploy-batch flush in >48h is a cheap, checkable rule”. Fleet directive T4 (mary-wiki docs/runbooks/2026-09-04-fleet-directive-remaining-blockers.md), same route wired in step in mary-wiki (mary-wiki#538) and recorded-not-built in panto-cloud (panto-cloud#15, no scheduled workflows there to watch).

A correction to that decision’s own record, found building this: deploy-batch did not, in fact, stay at “fired exactly once, ever” — it has run daily since 2026-08-31, five consecutive successful runs through 2026-09-04. Whatever caused the earlier drift-detection failure (hubble’s /api/deploy-drift 401 on private repos) either cleared or the cron itself recovered; either way, the “>48h” rule below is built as a standing check, not as a report that the historical claim still holds.

Decision

Section titled “Decision”

.github/workflows/scheduled-workflow-digest.yml, cron '30 8 * * *' (after deploy-batch’s 08:00 UTC slot) plus workflow_dispatch. Two checks, folded into one digest:

  1. GET /actions/runs?event=schedule&status=failure&created=>{24h-ago} — any scheduled workflow that failed in the last 24h.
  2. Age of the most recent completed deploy-batch run (any conclusion, not just failures — a dead cron produces no failure run either, only silence) — flagged if >48h.

Posts to the same WEBHOOK_PETROVA webhook main-red-alert.yml already uses — no new secret. Digest, not an immediate page, same reasoning as main-red-alert.yml’s own stated scope: scheduled jobs fail transiently and an immediate-fire route earns a muted channel inside a week. Posts only when there is something to report.

Verification

Section titled “Verification”

Falsified against real historical data, not a live forced break (no scheduled workflow can be triggered on demand with a genuine event=schedule — workflow_dispatch sets a different event and would not exercise the same filter honestly):

  • The failure query fires on real, currently-live data. As of this writing, sync-playbook-from-eva has five failure runs inside the actual last-24h window (33895094231, 33871600131 among them) — the exact query this workflow uses, run directly, returns them. This is not a synthetic case: it is the same live gap this ticket’s own context names (“sync-playbook-from-eva failed 2026-09-04, unnoticed”), still open at the time this digest was built. The first real run of this workflow will report it.
  • The staleness math is falsified on both a known-bad and the real known-good input. Simulated last_run_iso=2026-08-31T15:52:27Z (deploy-batch’s actual oldest recent run) against the current time yields age_hours=96, correctly >48 and firing. The real current last run (2026-09-04T12:31:53Z) yields age_hours=4, correctly not firing.

Not verified by this act: the live Discord POST path, unchanged from main-red-alert.yml’s already-falsified --fail-with-body mechanism (#399, same secret).

Sign-off

Section titled “Sign-off”