From 51d313357c93ab8303d8d26860fc206164ce7f55 Mon Sep 17 00:00:00 2001 From: mes123456 Date: Sun, 30 Aug 2026 14:05:10 +0800 Subject: [PATCH] fix(web): make continuous handoff self-recovering --- nextTask.md | 86 ++++++++------ tools/web/run-wbv2-continuous.mjs | 29 ++++- tools/web/run-wbv2-guardian.mjs | 189 ++++++++++++++++++++++++++++++ tools/web/run-wbv2-supervisor.mjs | 46 ++++++-- 4 files changed, 305 insertions(+), 45 deletions(-) create mode 100644 tools/web/run-wbv2-guardian.mjs diff --git a/nextTask.md b/nextTask.md index d8492c40d..1b49bd23c 100644 --- a/nextTask.md +++ b/nextTask.md @@ -57,40 +57,44 @@ capability family 和唯一行为批量推进 6900 条目，复用已有 hash 需要在终端保持进程级无人值守连续执行时，运行： ```bash -node tools/web/run-wbv2-continuous.mjs --max-iterations 0 --commit --push +node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push ``` `--max-iterations 0` 表示不限迭代次数；runner 只执行当前任务卡声明的 -入口，任务未推进、治理失败、构建失败或推送失败时立即停在原任务并保留日志。 -凭据必须使用已配置的 Git 凭据，不得写入命令行历史、URL、日志或仓库文件。 +入口。每次提交后必须同时满足 `git push` 成功和 `origin/main` 的远端 HEAD +等于本地 HEAD；网络抖动或远端确认延迟会自动退避重试，最终仍无法确认时保留 +精确 checkpoint。凭据必须使用已配置的 Git 凭据，不得写入命令行历史、URL、 +日志或仓库文件。 ## 进程级不间断执行 聊天消息只能启动一次代理执行；要跨越终端、网络或会话边界持续运行，必须 -启动仓库内 supervisor。supervisor 每次调用现有 runner 一个任务迭代，成功 -交接后立即启动下一迭代，并在重新启动时从 `docs/EXECUTION_QUEUE.md` 恢复， -不会依赖上一轮聊天上下文： +启动仓库内 guardian。guardian 持有唯一锁并托管 supervisor；supervisor 每次 +调用现有 runner 一个任务迭代，成功交接后立即启动下一迭代，并在重新启动时从 +`docs/EXECUTION_QUEUE.md` 恢复，不依赖上一轮聊天上下文： ```bash -node tools/web/run-wbv2-supervisor.mjs --max-iterations 0 --commit --push +node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push ``` 需要脱离终端运行时使用： ```bash -nohup node tools/web/run-wbv2-supervisor.mjs --max-iterations 0 --commit --push \ - > tests/golden/WBV2/continuous-supervisor.stdout.log 2>&1 < /dev/null & +setsid -f nohup node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push \ + > tests/golden/WBV2/continuous-guardian.stdout.log 2>&1 < /dev/null ``` -supervisor 的连续契约： +guardian/supervisor 的连续契约： - `--max-iterations 0` 表示不限任务轮数；每轮只执行当前 manifest 的一个 `nextTask`。 - 每轮先运行 `print-task-context` 和 `check-task-context`，再调用 runner；成功后立即重读队列。 - 每轮状态写入 `tests/golden/WBV2/continuous-supervisor.json`，详细输出写入 `continuous-supervisor.log`。 -- 使用锁文件防止多个 supervisor 同时改写队列；正常退出会清理锁文件。 -- 子进程失败、队列未前进、队列在失败后已经前进、推送失败或上下文不一致时立即保存 `blocked` checkpoint；不得盲目重试未知的状态变更。 -- supervisor 进程退出后，再次执行同一命令即可从当前队列恢复；不复制旧日志、不重放未知命令。 -- supervisor 只负责有效 handoff 的连续执行，不能绕过真实阻塞、治理门禁、源码编译失败或 provenance 规则。 +- guardian 状态写入 `tests/golden/WBV2/continuous-guardian.json`，详细输出写入 `continuous-guardian.log`；两个锁文件均防止重复实例。 +- runner 的 push 流程最多退避重试 6 次，并在每次提交后执行 `git ls-remote origin refs/heads/main` 对比本地 HEAD；未确认远端提交不得进入下一任务。 +- supervisor 意外退出时，guardian 仅当 checkpoint 为 `handed_off` 且当前队列与 checkpoint.after 完全一致时自动拉起；这是唯一可证明不会重放未知写操作的恢复路径。 +- 子进程失败、队列未前进、队列在失败后已经前进、推送无法确认或上下文不一致时保存 `blocked` checkpoint；不得盲目重试未知的状态变更。 +- guardian 或 supervisor 进程退出后，再次执行同一 guardian 命令即可从当前队列恢复；不复制旧日志、不重放未知命令。 +- guardian 只负责有效 handoff 和可证明的进程恢复，不能绕过真实阻塞、治理门禁、源码编译失败或 provenance 规则。 ## Copyable Resume Prompt @@ -113,14 +117,16 @@ and (when authorized) repository push: 6. 原始 P0 inventory、6900 总数、6618 legacy pending 和历史 provenance 始终只读。effective completed/pending 只由后续 hash-bound 验证 overlay 计算；实现完成不等于条目完成。 7. 长输出写入 `tests/golden//command.log`；交互只报告退出码、短尾部、构件路径和 SHA-256。 8. 任务成功后运行 `node tools/web/check-task-context.mjs --task --write`、`node tools/web/check-context-governance.mjs`、`node tools/web/check-v2-task-index.mjs`、`npm --prefix web run typecheck`、`npm --prefix web run lint` 和 `git diff --check`；完整生产验证门禁统一在验证波次执行。 -9. 所有实现门禁通过后自动提交并推送 `origin/main`，从新 parent manifest 重新读取 `nextTask` 继续。凭据不得写入文件、URL、日志或提示词。 +9. 所有实现门禁通过后自动提交并推送 `origin/main`；必须再次用 `git ls-remote` 确认远端 `main` 与本地 `HEAD` 完全相同，确认完成后才从新 parent manifest 读取 `nextTask` 继续。凭据不得写入文件、URL、日志或提示词。 10. 任一命令失败、输入缺失、队列不一致、环境不可用或状态不明时，保持 `in_progress`/`blocked`，不得推进指针、伪造完成或重复未知结果的状态改变命令。 11. 每轮结束报告实现切片、当前任务、`nextTask`、最小检查退出码、原始 inventory、effective inventory 是否保持不变、提交/推送状态和剩余真实阻塞。 12. CONTINUOUS_LOOP_REQUIRED：成功交接后立即开始下一轮；不得在成功提交、推送或生成新 manifest 后等待用户确认、只写计划或只汇报状态。每轮必须重新运行 `print-task-context` 和 `check-task-context`，使用新 parent manifest 的 `nextTask`，不沿用上一轮上下文。 -13. 可使用仓库内 unattended runner 保持进程级连续执行：`node tools/web/run-wbv2-continuous.mjs --max-iterations 0 --commit --push`。它只执行任务卡声明的入口；`0` 表示不限轮数，`--max-iterations N` 用于受控运行，`--commit --push` 才允许自动提交/推送。任何入口失败、退出 0 但队列未前进、治理失败或 push 失败都立即停止并保留 `tests/golden//continuous-runner.log`，不得重试未知结果。 +13. 可使用仓库内 guardian 保持进程级连续执行：`node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push`。它托管 supervisor，`0` 表示不限轮数，`--max-iterations N` 用于受控运行，`--commit --push` 才允许自动提交/推送。push 网络抖动由 runner 自动退避重试并确认远端 HEAD；任何入口失败、退出 0 但队列未前进、治理失败、远端无法确认或状态不明都保留 checkpoint，guardian 只自动恢复可证明安全的 handoff。 14. 6900_IMPLEMENTATION_IRON_RULE：只按 capability family 和唯一行为批次实现全部 Blender 功能；不启动 Chromium/E2E/receipt/parity/save-reopen/determinism 验证，不写 completion overlay，不把维护 smoke 当作能力完成。若任务卡与本铁律冲突，以源码实现和本铁律为准，并将验证留到后续独立验证波次。 15. UNIFIED_WASM_JSON_ABI_IRON_RULE：Worker 与 WASM 的新调用统一使用单一 C ABI `web_engine_call_json(handle, requestJson, resultOut)`；请求必须是 `{"cmd":"...","params":{...}}`，返回必须是 `{"ok":true,"cmd":"...","result":...}` 或包含 `code/error` 的失败 JSON，再由 Worker 解析为领域结果。`applyCommand`、快照、增量、撤销/重做、Depsgraph 等状态路径不得新增独立公开 WASM 函数；open/append/save/geometry 等仍未迁移的二进制专用 ABI 仅作为兼容边界保留，并不得扩散到新的状态调用。 -16. PROCESS_CONTINUITY_IRON_RULE：要实现跨会话不间断执行，必须使用 `run-wbv2-supervisor.mjs --max-iterations 0`；单次聊天执行完成后不得假定代理会在下一轮自动复活。supervisor 必须按任务原子迭代、锁队列、写 checkpoint、成功后立即重读 manifest；失败或状态未知时只能停在精确 checkpoint，不得自动重试或伪造队列前进。 +16. PROCESS_CONTINUITY_IRON_RULE：要实现跨会话不间断执行，必须使用 `run-wbv2-guardian.mjs --max-iterations 0` 托管 `run-wbv2-supervisor.mjs`；单次聊天执行完成后不得假定代理会在下一轮自动复活。guardian/supervisor 必须按任务原子迭代、锁队列、写 checkpoint、成功后立即重读 manifest；失败或状态未知时只能停在精确 checkpoint，不得自动重试或伪造队列前进。 +17. PUSH_COMPLETION_IRON_RULE：任何任务只有在 commit 成功、`git push origin main` 成功且 `git ls-remote --refs origin refs/heads/main` 返回的远端 HEAD 与本地 `git rev-parse HEAD` 相同后才算完成；确认前不得读取下一任务。网络失败最多按退避策略重试，状态未知时不得重放任务写入。 +18. GUARDIAN_CONTINUITY_IRON_RULE：无人值守运行必须由 `run-wbv2-guardian.mjs` 托管 supervisor。guardian 对可证明的 `handed_off` checkpoint 自动恢复，对 `running`、`started`、`blocked`、指针漂移、远端提交不明或任务闭合状态保持精确停止；“不停歇”仅适用于安全可恢复的进程/网络故障，不能解释为绕过真实阻塞。 ``` ## Resume Command @@ -150,8 +156,10 @@ review, or diagnosis do not activate this procedure. There is no intentional idle step between valid handoffs. Each iteration still has exactly one numbered task and must not carry the previous task's full -transcript, logs, or source context. A continuous run may contain many such -iterations, but it must stop at the first failed gate or explicit stop request. +transcript, logs, or source context. A guardian run may contain many such +iterations and automatically recovers process exits after a proven handoff; +it stops only at a real gate failure, unknown state, queue closure, or explicit +stop request. ## Continuous Loop Contract @@ -188,12 +196,14 @@ not completion overlays. For process-level unattended execution, run: ```bash -node tools/web/run-wbv2-continuous.mjs --max-iterations 0 --commit --push +node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push ``` The runner is intentionally fail-closed. It derives the command from the current task card, requires the queue to advance after a successful exit, and -writes one bounded log per task. It never invents a task, retries an unknown +writes one bounded log per task. Push completion is accepted only after the +remote `origin/main` HEAD equals the local commit. The guardian retries only a +proven `handed_off` process exit; it never invents a task, retries an unknown state-changing result, or turns implementation output into parity completion. ## Activation Contract @@ -224,9 +234,12 @@ hashes, and governance checks agree. A task that fails, is cancelled, exceeds budget, lacks its environment, or has hash drift remains `in_progress` or `blocked`; the next invocation resumes that same task instead of advancing. -## Stop Conditions +## Stop Conditions And Recovery -Continuous mode must stop immediately when any of these conditions occurs: +The unattended guardian does not stop for a transient process or push transport +failure. The runner retries push/remote-HEAD confirmation with bounded +exponential backoff, and the guardian restarts only after a proven handoff. +Continuous mode stops only when an unsafe or irreversible condition remains: - a focused command, exit criterion, governance check, or hash check fails; - the queue, parent manifest, task card, task index, or `nextTask` pointer is @@ -234,14 +247,16 @@ Continuous mode must stop immediately when any of these conditions occurs: - the environment is missing, the task is cancelled, or a context/size budget would be exceeded; - a stream disconnect, remote compact error, or other transport error makes the - result of a state-changing command unknown; + result of a state-changing command unknown after read-only reconciliation; - the queue has no next task or reaches its declared closure gate; - the user explicitly asks to stop or pause. -On a stop, preserve the current checkpoint, do not start another task, and -report the exact task, state, failed gate or stop reason, and evidence paths. -Only a successful handoff starts another iteration. There is no retry loop for -an unknown or failed result. +On an unsafe stop, preserve the current checkpoint, do not start another task, +and report the exact task, state, failed gate or stop reason, and evidence paths. +Only a successful handoff starts another iteration. A process exit is not by +itself a reason to stop: the guardian re-enters only from a matching +`handed_off` checkpoint; `running`/`started` or mismatched pointers remain +blocked to prevent duplicate writes. For every task, run this exact order: @@ -285,9 +300,10 @@ to the newly completed manifest pointer. ## Transport/Compaction Recovery `stream disconnected`, `error decoding response body`, or a remote compact -failure means that the request outcome is unknown. It is not evidence that the -task failed or succeeded. Do not blindly repeat a generator, build, save, or -queue-writing command. +failure means that the request outcome is unknown. The process-level guardian +continues independently when its checkpoint is valid; a chat client must first +perform read-only reconciliation. Do not blindly repeat a generator, build, +save, or queue-writing command. Recover with read-only checks first: @@ -301,9 +317,11 @@ Then compare the current task's manifest, status, task-context, and artifact hashes. Continue from the first missing handoff step; if every criterion is already recorded, run only the missing governance/check command. The failed request's outcome is unknown, so do not claim success from the error message. -Resume the request at most once. If the resumed request disconnects again, -leave the task `in_progress`/`blocked`, preserve all files, and report the exact -command and checkpoint state rather than starting a retry loop. +Resume the request at most once in a chat session. For unattended execution, +push transport retries are handled by the runner and proven-handoff process +restarts by the guardian; if reconciliation cannot prove safety, leave the +task `in_progress`/`blocked`, preserve all files, and report the exact command +and checkpoint state rather than starting an unsafe retry loop. If the client supplies a session id, `codex resume ` may be used, but the resumed turn must still reread the queue and print a fresh context. Do diff --git a/tools/web/run-wbv2-continuous.mjs b/tools/web/run-wbv2-continuous.mjs index 031d62f3a..c87480bcd 100644 --- a/tools/web/run-wbv2-continuous.mjs +++ b/tools/web/run-wbv2-continuous.mjs @@ -11,8 +11,18 @@ const maxIterations = maxArg >= 0 ? Number(process.argv[maxArg + 1]) : 0; if (!Number.isSafeInteger(maxIterations) || maxIterations < 0) throw new Error("--max-iterations must be a non-negative integer; 0 means unlimited"); const commitEnabled = process.argv.includes("--commit"); const pushEnabled = process.argv.includes("--push"); +if (pushEnabled && !commitEnabled) throw new Error("--push requires --commit"); +const pushRetryArg = process.argv.indexOf("--push-retries"); +const pushRetries = pushRetryArg >= 0 ? Number(process.argv[pushRetryArg + 1]) : 6; +if (!Number.isSafeInteger(pushRetries) || pushRetries < 0) throw new Error("--push-retries must be a non-negative integer"); const shellQuote = (value) => `'${value.replaceAll("'", "'\\''")}'`; +function waitMs(milliseconds) { + if (milliseconds <= 0) return; + const wait = new Int32Array(new SharedArrayBuffer(4)); + Atomics.wait(wait, 0, 0, milliseconds); +} + function command(command, logPath) { const result = spawnSync("bash", ["-lc", command], { cwd: root, encoding: "utf8", maxBuffer: 16 * 1024 * 1024 }); const output = `${result.stdout ?? ""}${result.stderr ?? ""}`; @@ -78,8 +88,23 @@ while (maxIterations === 0 || iterations < maxIterations) { const commitResult = command(`git diff --cached --quiet || git commit -m "chore(web): continuous handoff ${before.task}"`, `tests/golden/${before.task}/continuous-commit-result.log`); if (commitResult.code !== 0) throw new Error(`${before.task} commit failed; log=tests/golden/${before.task}/continuous-commit-result.log`); if (pushEnabled) { - const push = command("git push origin main", `tests/golden/${before.task}/continuous-push.log`); - if (push.code !== 0) throw new Error(`${before.task} push failed; log=tests/golden/${before.task}/continuous-push.log`); + let pushed = false; + for (let attempt = 0; attempt <= pushRetries; attempt += 1) { + const pushLog = attempt === 0 + ? `tests/golden/${before.task}/continuous-push.log` + : `tests/golden/${before.task}/continuous-push-attempt-${attempt + 1}.log`; + const push = command("git push origin main", pushLog); + const pushVerification = command( + "local_head=$(git rev-parse --verify HEAD) && remote_head=$(git ls-remote --refs origin refs/heads/main | awk '$2 == \"refs/heads/main\" { print $1; exit }') && printf 'local_head=%s remote_head=%s\\n' \"$local_head\" \"$remote_head\" && test -n \"$remote_head\" && test \"$local_head\" = \"$remote_head\"", + `tests/golden/${before.task}/continuous-push-verify.log`, + ); + if (pushVerification.code === 0) { + pushed = true; + break; + } + if (attempt < pushRetries) waitMs(Math.min(30000, 1000 * 2 ** attempt)); + } + if (!pushed) throw new Error(`${before.task} remote push verification failed after ${pushRetries + 1} attempts; log=tests/golden/${before.task}/continuous-push-verify.log`); } } } diff --git a/tools/web/run-wbv2-guardian.mjs b/tools/web/run-wbv2-guardian.mjs new file mode 100644 index 000000000..f9f1a4d6b --- /dev/null +++ b/tools/web/run-wbv2-guardian.mjs @@ -0,0 +1,189 @@ +import fs from "node:fs"; +import path from "node:path"; +import { spawn } from "node:child_process"; +import { fileURLToPath } from "node:url"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); +const abs = (file) => path.join(root, file); +const controlRoot = "tests/golden/WBV2"; +const logPath = `${controlRoot}/continuous-guardian.log`; +const checkpointPath = `${controlRoot}/continuous-guardian.json`; +const lockPath = `${controlRoot}/continuous-guardian.lock`; + +const argv = process.argv.slice(2); +const numericArg = (name, fallback) => { + const index = argv.indexOf(name); + return Number(index >= 0 ? argv[index + 1] : fallback); +}; +const restartDelayMs = numericArg("--restart-delay-ms", 5000); +const maxRestarts = numericArg("--max-restarts", 0); +const maxIterations = numericArg("--max-iterations", 0); +if (!Number.isSafeInteger(restartDelayMs) || restartDelayMs < 0) throw new Error("--restart-delay-ms must be a non-negative integer"); +if (!Number.isSafeInteger(maxRestarts) || maxRestarts < 0) throw new Error("--max-restarts must be a non-negative integer; 0 means unlimited safe restarts"); +if (!Number.isSafeInteger(maxIterations) || maxIterations < 0) throw new Error("--max-iterations must be a non-negative integer; 0 means unlimited"); +if (argv.includes("--push") && !argv.includes("--commit")) throw new Error("--push requires --commit"); + +const supervisorArgs = []; +for (let index = 0; index < argv.length; index += 1) { + if (argv[index] === "--restart-delay-ms" || argv[index] === "--max-restarts") { + index += 1; + continue; + } + supervisorArgs.push(argv[index]); +} +const supervisorCommand = ["tools/web/run-wbv2-supervisor.mjs", ...supervisorArgs].join(" "); +const supervisorNodeArgs = ["tools/web/run-wbv2-supervisor.mjs", ...supervisorArgs]; + +fs.mkdirSync(abs(controlRoot), { recursive: true }); +let lockHandle; +const acquireLock = () => { + try { + return fs.openSync(abs(lockPath), "wx"); + } + catch (error) { + if (error?.code !== "EEXIST") throw error; + try { + const lock = JSON.parse(fs.readFileSync(abs(lockPath), "utf8")); + if (Number.isSafeInteger(lock.pid)) { + try { process.kill(lock.pid, 0); } + catch (probeError) { + if (probeError?.code === "ESRCH") { + fs.unlinkSync(abs(lockPath)); + return fs.openSync(abs(lockPath), "wx"); + } + } + } + } + catch { + // An unreadable lock is treated as owned to avoid concurrent guardians. + } + throw new Error(`another WBV2 guardian appears to be running; lock=${lockPath}`); + } +}; +lockHandle = acquireLock(); +fs.writeFileSync(lockHandle, `${JSON.stringify({ pid: process.pid, startedAt: new Date().toISOString() })}\n`); + +let released = false; +const releaseLock = () => { + if (released || lockHandle === undefined) return; + released = true; + fs.closeSync(lockHandle); + try { fs.unlinkSync(abs(lockPath)); } + catch (error) { if (error?.code !== "ENOENT") throw error; } +}; +const appendLog = (text) => fs.appendFileSync(abs(logPath), `[${new Date().toISOString()}] ${text}\n`); +const writeCheckpoint = (payload) => fs.writeFileSync(abs(checkpointPath), `${JSON.stringify({ schemaVersion: 1, pid: process.pid, updatedAt: new Date().toISOString(), ...payload }, null, 2)}\n`); +const readJson = (file) => { + try { return JSON.parse(fs.readFileSync(abs(file), "utf8")); } + catch { return null; } +}; +const currentTask = () => { + const queue = fs.readFileSync(abs("docs/EXECUTION_QUEUE.md"), "utf8"); + const task = queue.match(/\| 当前任务 \| `(WBV2-P\d+-\d+)` \|/u)?.[1]; + const parentManifest = queue.match(/\| parent manifest \| `([^`]+)` \|/u)?.[1]; + if (!task || !parentManifest) throw new Error("queue pointer is incomplete"); + return { task, parentManifest }; +}; +const samePointer = (left, right) => left?.task === right?.task && left?.parentManifest === right?.parentManifest; +const sleep = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)); + +let child; +let stopRequested = false; +let stopSignal; +const signalChild = (signal) => { + if (!child?.pid) return; + try { process.kill(-child.pid, signal); } + catch (error) { + if (error?.code !== "ESRCH") { + try { child.kill(signal); } + catch { /* The exit handler will record the final state. */ } + } + } +}; +const requestStop = (signal) => { + stopRequested = true; + stopSignal = signal; + appendLog(`stop-requested signal=${signal}`); + signalChild(signal); +}; +process.once("SIGINT", () => requestStop("SIGINT")); +process.once("SIGTERM", () => requestStop("SIGTERM")); + +const runSupervisor = () => new Promise((resolve) => { + child = spawn(process.execPath, supervisorNodeArgs, { + cwd: root, + detached: true, + env: process.env, + stdio: "inherit", + }); + child.once("error", (error) => resolve({ code: 1, signal: null, error: error.message })); + child.once("exit", (code, signal) => { + child = undefined; + resolve({ code: code ?? 1, signal }); + }); +}); + +function safeRestart(checkpoint, pointer) { + if (!checkpoint) return { ok: false, reason: "supervisor_checkpoint_missing" }; + if (checkpoint.state === "blocked") return { ok: false, reason: checkpoint.reason ?? "supervisor_blocked" }; + if (checkpoint.state === "stopped") return { ok: false, reason: checkpoint.reason ?? "supervisor_stopped" }; + if (checkpoint.state !== "handed_off") return { ok: false, reason: `supervisor_state_${checkpoint.state ?? "unknown"}` }; + if (!samePointer(checkpoint.after, pointer)) return { ok: false, reason: "handoff_pointer_mismatch" }; + return { ok: true }; +} + +let guardianExit = 0; +let restartCount = 0; +try { + appendLog(`started pid=${process.pid} supervisor=${supervisorCommand} maxRestarts=${maxRestarts} restartDelayMs=${restartDelayMs}`); + writeCheckpoint({ state: "started", restartCount, supervisor: supervisorCommand, current: currentTask() }); + while (true) { + if (stopRequested) { + writeCheckpoint({ state: "stopped", restartCount, reason: stopSignal, current: currentTask() }); + guardianExit = stopSignal === "SIGINT" ? 130 : 143; + break; + } + const before = currentTask(); + writeCheckpoint({ state: "running", restartCount, before, supervisor: supervisorCommand }); + const result = await runSupervisor(); + const checkpoint = readJson(checkpointPath); + const after = currentTask(); + appendLog(`supervisor-exit code=${result.code} signal=${result.signal ?? "none"} task=${before.task} queue=${after.task} state=${checkpoint?.state ?? "missing"}`); + if (stopRequested) { + writeCheckpoint({ state: "stopped", restartCount, reason: stopSignal, before, after, supervisorExit: result.code }); + guardianExit = stopSignal === "SIGINT" ? 130 : 143; + break; + } + if (checkpoint?.state === "complete") { + writeCheckpoint({ state: "complete", restartCount, current: after, supervisorExit: result.code }); + break; + } + const decision = safeRestart(checkpoint, after); + if (!decision.ok) { + writeCheckpoint({ state: "blocked", restartCount, before, after, reason: decision.reason, supervisorExit: result.code }); + appendLog(`blocked reason=${decision.reason}; no automatic replay`); + guardianExit = 2; + break; + } + restartCount += 1; + if (maxRestarts > 0 && restartCount > maxRestarts) { + writeCheckpoint({ state: "blocked", restartCount, before, after, reason: "max_safe_restarts_exceeded", supervisorExit: result.code }); + appendLog("blocked reason=max_safe_restarts_exceeded"); + guardianExit = 3; + break; + } + writeCheckpoint({ state: "restarting", restartCount, after, reason: "safe_handoff_checkpoint" }); + appendLog(`restarting count=${restartCount} after=${after.task}`); + await sleep(restartDelayMs); + } +} +catch (error) { + guardianExit = 4; + writeCheckpoint({ state: "blocked", restartCount, reason: "guardian_error", error: error?.message ?? String(error) }); + appendLog(`blocked reason=guardian_error error=${error?.message ?? String(error)}`); +} +finally { + signalChild("SIGTERM"); + releaseLock(); +} +process.exitCode = guardianExit; diff --git a/tools/web/run-wbv2-supervisor.mjs b/tools/web/run-wbv2-supervisor.mjs index d9f5e0b11..da9664259 100644 --- a/tools/web/run-wbv2-supervisor.mjs +++ b/tools/web/run-wbv2-supervisor.mjs @@ -16,10 +16,12 @@ const valueArg = (name, fallback) => { }; const maxIterations = Number(valueArg("--max-iterations", "0")); const pollMs = Number(valueArg("--poll-ms", "1000")); +const pushRetries = Number(valueArg("--push-retries", "6")); const commitEnabled = process.argv.includes("--commit"); const pushEnabled = process.argv.includes("--push"); if (!Number.isSafeInteger(maxIterations) || maxIterations < 0) throw new Error("--max-iterations must be a non-negative integer; 0 means unlimited"); if (!Number.isSafeInteger(pollMs) || pollMs < 0) throw new Error("--poll-ms must be a non-negative integer"); +if (!Number.isSafeInteger(pushRetries) || pushRetries < 0) throw new Error("--push-retries must be a non-negative integer"); if (pushEnabled && !commitEnabled) throw new Error("--push requires --commit"); fs.mkdirSync(abs(supervisorRoot), { recursive: true }); @@ -59,19 +61,34 @@ const releaseLock = () => { try { fs.unlinkSync(abs(lockPath)); } catch (error) { if (error?.code !== "ENOENT") throw error; } }; +let stopRequested = false; +let stopSignal; +let iteration = 0; +let fatalHandled = false; const stopFromSignal = (signal) => { - try { - writeCheckpoint({ state: "stopped", iteration: 0, reason: signal }); - appendLog(`stopped signal=${signal}`); - releaseLock(); - } - finally { - process.exitCode = signal === "SIGINT" ? 130 : 143; - } + stopRequested = true; + stopSignal = signal; + appendLog(`stop-requested signal=${signal}`); + process.exitCode = signal === "SIGINT" ? 130 : 143; }; process.once("SIGINT", () => stopFromSignal("SIGINT")); process.once("SIGTERM", () => stopFromSignal("SIGTERM")); +const stopOnFatal = (error, source) => { + if (fatalHandled) return; + fatalHandled = true; + try { + writeCheckpoint({ state: "blocked", iteration, reason: source, error: error?.message ?? String(error) }); + appendLog(`blocked reason=${source} error=${error?.message ?? String(error)}`); + } + finally { + releaseLock(); + process.exit(1); + } +}; +process.once("uncaughtException", (error) => stopOnFatal(error, "uncaught_exception")); +process.once("unhandledRejection", (error) => stopOnFatal(error, "unhandled_rejection")); + if (lockHandle === undefined) { throw new Error(`unable to acquire supervisor lock=${lockPath}`); } @@ -96,14 +113,18 @@ const currentTask = () => { const childArgs = ["tools/web/run-wbv2-continuous.mjs", "--max-iterations", "1"]; if (commitEnabled) childArgs.push("--commit"); if (pushEnabled) childArgs.push("--push"); +if (pushEnabled) childArgs.push("--push-retries", String(pushRetries)); appendLog(`started pid=${process.pid} maxIterations=${maxIterations} child=${childArgs.join(" ")}`); writeCheckpoint({ state: "started", iteration: 0, current: currentTask(), child: childArgs.join(" ") }); -let iteration = 0; let exitCode = 0; try { while (maxIterations === 0 || iteration < maxIterations) { + if (stopRequested) { + writeCheckpoint({ state: "stopped", iteration, current: currentTask(), reason: stopSignal }); + break; + } const before = currentTask(); const print = run(process.execPath, ["tools/web/print-task-context.mjs"]); const check = run(process.execPath, ["tools/web/check-task-context.mjs"]); @@ -117,6 +138,10 @@ try { const child = run(process.execPath, childArgs); appendLog(`child task=${before.task} exit=${child.code}\n${child.output}`); const after = currentTask(); + if (stopRequested) { + writeCheckpoint({ state: "stopped", iteration, before, after, reason: stopSignal, childExit: child.code }); + break; + } if (child.code !== 0) { const reason = after.task === before.task ? "child_failed_same_task" : "child_failed_after_queue_advance"; writeCheckpoint({ state: "blocked", iteration: iteration + 1, before, after, reason, childExit: child.code }); @@ -136,6 +161,9 @@ try { Atomics.wait(wait, 0, 0, pollMs); } } + if (stopRequested) { + exitCode = stopSignal === "SIGINT" ? 130 : 143; + } if (exitCode === 0) { const current = currentTask(); writeCheckpoint({ state: maxIterations === 0 ? "running" : "complete", iteration, current });