Skip to content

The weekly IRINA cycle deploys the apex after it records history, and the dead-man push is retired


The weekly IRINA cycle deploys the apex after it records history, and the dead-man push is retired

Section titled “The weekly IRINA cycle deploys the apex after it records history, and the dead-man push is retired”

Date: 2026-11-04 Status: closed Supersedes: docs/decisions/2026-08-31-host-production-deploys-move-to-the-hook.md (partial: only its consequence that merging to main publishes nothing, so the weekly IRINA history never reaches the apex. The hook, the disabled git deploys and the cost reasoning stand.) Superseded-by: none — current MR-compliance: MR-4, MR-7

RS-DL-003 R-07 needs the IRINA history-freshness check green. Its mcp row passes. Its apex row (petrova.host) fails every week: run 41705614559 (2026-11-03T19:41:31Z, failure) reports the apex newest entry as 2026-10-05T18:09:50.362Z against a main newest of 2026-11-02T19:14:10.870Z, a lag of 2423061s against the 648000s limit. Commit 394bce26 (the ignored-build predicate fix) changed nothing. Ticket FT-TL-004 asked for the cause to be proved by observation.

Observations, all read on 2026-11-04 (UTC):

  • Written. Main carries the history up to 2026-11-02T19:14:10.870Z (commit 9adbbfee). Hypothesis “apex history is not written” is falsified.
  • Not deployed. The newest READY production deployment of petrova-host (project prj_YzYyIAYuaQ7GJQh1HBSYAidg2uZe) is dpl_HKzRiXaVvcn1sxmZoZ1i5WkqhW3k, created 2026-10-10T14:21:30Z (Vercel list_deployments, target production, read 2026-11-04T08:3xZ). The apex newest entry (2026-10-05) is the last cycle before it. Nothing newer exists in the production list.
  • Mechanism. vercel.json at 6e79758f carries git.deploymentEnabled.main = false (decision 2026-08-31). The only production path is the deploy hook, fired by deploy-batch.yml only for projects hubble reports as drifted. Run 41668197492 (2026-11-03) reports 8 drifted projects of 68, none of them petrova-host, and posted no hook. The 2026-08-31 decision records that hubble answers 401 for private repos, so petrova-host reports unknown. The ignored-build predicate is never reached when no git deploy starts, which is why 394bce26 changed nothing.
  • Checker and reader are right. The mcp row and the control arm pass with the same code (host/tests/irina-history-freshness.test.ts, extended in this change).

Not observable from this session, stated as unknown: the live apex answer (needs LIVENESS_APEX_BEARER), hubble’s current status for petrova-host (needs HUBBLE_INGEST_TOKEN), whether the DSN pull rules (devarno-cloud/deep-space-network fa18c98f) are deployed (tailnet), and whether the value of VERCEL_DEPLOY_HOOK_PETROVA_HOST is a working hook (secrets are never read by agents). None changes the decision below.

The dead-man push step in record-history warns on every run on the hosted runner. Run 41507879030 (2026-11-02) logged push: HTTP 000 on a GitHub-hosted runner. The one earlier 2xx, run 40353694874 (2026-10-26, push: HTTP 204), ran on the self-hosted runner hermes-fleet-1, before the 2026-10-30 flip of this workflow to ubuntu-latest (44480610). Decision 2026-10-24 (B-8) records that a hosted runner cannot reach the tailnet target. Since the flip, the step cannot succeed.

  1. record-history in irina-cycle.yml ends with a step that POSTs the existing petrova-host deploy hook (secrets.VERCEL_DEPLOY_HOOK_PETROVA_HOST) after the history commit has been pushed. It runs on the no-change path too, has no continue-on-error, fails on an unset secret or a non-2xx answer, and so turns the cycle red (which main-red-alert already watches). The apex is deployed once a week, exactly when the baked data changes.
  2. The “Dead-man ping” step is removed, not converted to fail-the-job. Failing the job on a cause that never clears on a hosted runner would be permanent red and train the operator to ignore main-red-alert, which is another way to absorb a permanent failure. Nothing is lost: the step never delivered from a hosted runner, and the DSN pull rule (evidence-freshness, reading the newest scanned_at through the GitHub API) is the signal. irina/deadman-exposition.mjs stays, because the pull collector and host/scripts/irina-history-freshness.mjs use its parse rule.
  3. A test pins that no step in irina-cycle.yml may combine continue-on-error: true with an exit 1.
  4. MAX_LAG_SECONDS stays 648000, now pinned by a test so that a loosening shows in review.

OC-3 of decision 2026-10-24 (how a push would reach the plane) is answered by the pull route: its options (a) and (c) are not needed. The operator may reject this at countersign.

  • Re-enable git deploys of main — reverses the whole 2026-08-31 decision and brings back the volume it removed (422 commits on main since 2026-10-05).
  • Dispatch deploy-batch with only_project=petrova-host — it still gates on hubble’s drift status, which is unknown for this project, so it posts nothing.
  • Fix hubble /api/deploy-drift for private repos — the right durable repair for every governed project, but another repo and owner. Follow-up, not a dependency.
  • Deploy from inside irina-history-freshness.yml — a check that deploys repairs its own failure and so can never fail.
  • Read history at request time from GitHub — needs a runtime credential in the function and a larger host change.
  • Raise MAX_LAG_SECONDS — forbidden by the ticket; it would hide the defect.
  • Make the dead-man push fail the job — permanent red on a hosted runner; see Decision 2.
  • Keep the push as a warning — a guard absorbing a permanent failure.

For code:

  • .github/workflows/irina-cycle.yml: dead-man step removed, apex deploy-hook step added.
  • irina/tests/workflow-pins.test.ts: the pin that required continue-on-error: true on the dead-man step is replaced by pins for its absence, the silent-warning shape and the deploy step.
  • host/tests/irina-history-freshness.test.ts: apex-staleness cases at the observed values.

For docs:

  • Watch item: any future surface that bakes history at build time while its git deploys are off shares the apex’s blind spot. SURFACES holds only mcp and apex today. Not built.
  • Operator acts owed after merge: confirm the hook secret holds a working URL; fire the hook once and dispatch IRINA history freshness (OP-TL-05 act 3); optionally delete the now-inert DEADMAN_PUSH_URL repository variable.

For in-flight phases:

  • RS-DL-003 R-07a (freshness green) is met only when a dispatched freshness run is green on both rows after a deploy. This change does not deploy anything.

For invariants:

  • MR-7: decision 2026-08-31 is not edited; this forward decision supersedes one consequence.
  • docs/decisions/2026-08-31-host-production-deploys-move-to-the-hook.md
  • docs/decisions/2026-10-24-ci-off-eu1-hosted-nightly.md (B-8, OC-3)
  • Runs: 41705614559 (freshness, 2026-11-03), 41507879030 (IRINA cycle, 2026-11-02, HTTP 000), 40353694874 (IRINA cycle, 2026-10-26, HTTP 204 on hermes-fleet-1), 41668197492 (deploy-batch, 2026-11-03)
  • devarno-cloud/deep-space-network fa18c98f (DSN pull rules)
  • devarno-cloud ticket FT-TL-004 (run 2026-11-04)
  • Subagent: tl-builder (trust-loop fleet, FT-TL-004, run 2026-11-04)
  • Human countersign — the weekly cycle deploying the apex through the hook, the dead-man push retired in favour of the DSN pull, and OC-3 options (a) and (c) dropped — flight@arno.host, 2026-11-04 (jzero.run T1)

Countersigned by human:flight@arno.host on 2026-11-04 — box ticked in commit 211650b3, authored by the human (jzero.run T1, session alex@devarno.cloud): the weekly cycle deploying the apex through the hook, the dead-man push retired in favour of the DSN pull, and OC-3 options (a) and (c) dropped. Status flipped open→closed on 2026-11-04 by an agent as scribe (MR-7).