fix(web): make continuous handoff self-recovering
This commit is contained in:
86
nextTask.md
86
nextTask.md
@@ -57,40 +57,44 @@ capability family 和唯一行为批量推进 6900 条目,复用已有 hash
|
||||
需要在终端保持进程级无人值守连续执行时,运行:
|
||||
|
||||
```bash
|
||||
node tools/web/run-wbv2-continuous.mjs --max-iterations 0 --commit --push
|
||||
node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push
|
||||
```
|
||||
|
||||
`--max-iterations 0` 表示不限迭代次数;runner 只执行当前任务卡声明的
|
||||
入口,任务未推进、治理失败、构建失败或推送失败时立即停在原任务并保留日志。
|
||||
凭据必须使用已配置的 Git 凭据,不得写入命令行历史、URL、日志或仓库文件。
|
||||
入口。每次提交后必须同时满足 `git push` 成功和 `origin/main` 的远端 HEAD
|
||||
等于本地 HEAD;网络抖动或远端确认延迟会自动退避重试,最终仍无法确认时保留
|
||||
精确 checkpoint。凭据必须使用已配置的 Git 凭据,不得写入命令行历史、URL、
|
||||
日志或仓库文件。
|
||||
|
||||
## 进程级不间断执行
|
||||
|
||||
聊天消息只能启动一次代理执行;要跨越终端、网络或会话边界持续运行,必须
|
||||
启动仓库内 supervisor。supervisor 每次调用现有 runner 一个任务迭代,成功
|
||||
交接后立即启动下一迭代,并在重新启动时从 `docs/EXECUTION_QUEUE.md` 恢复,
|
||||
不会依赖上一轮聊天上下文:
|
||||
启动仓库内 guardian。guardian 持有唯一锁并托管 supervisor;supervisor 每次
|
||||
调用现有 runner 一个任务迭代,成功交接后立即启动下一迭代,并在重新启动时从
|
||||
`docs/EXECUTION_QUEUE.md` 恢复,不依赖上一轮聊天上下文:
|
||||
|
||||
```bash
|
||||
node tools/web/run-wbv2-supervisor.mjs --max-iterations 0 --commit --push
|
||||
node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push
|
||||
```
|
||||
|
||||
需要脱离终端运行时使用:
|
||||
|
||||
```bash
|
||||
nohup node tools/web/run-wbv2-supervisor.mjs --max-iterations 0 --commit --push \
|
||||
> tests/golden/WBV2/continuous-supervisor.stdout.log 2>&1 < /dev/null &
|
||||
setsid -f nohup node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push \
|
||||
> tests/golden/WBV2/continuous-guardian.stdout.log 2>&1 < /dev/null
|
||||
```
|
||||
|
||||
supervisor 的连续契约:
|
||||
guardian/supervisor 的连续契约:
|
||||
|
||||
- `--max-iterations 0` 表示不限任务轮数;每轮只执行当前 manifest 的一个 `nextTask`。
|
||||
- 每轮先运行 `print-task-context` 和 `check-task-context`,再调用 runner;成功后立即重读队列。
|
||||
- 每轮状态写入 `tests/golden/WBV2/continuous-supervisor.json`,详细输出写入 `continuous-supervisor.log`。
|
||||
- 使用锁文件防止多个 supervisor 同时改写队列;正常退出会清理锁文件。
|
||||
- 子进程失败、队列未前进、队列在失败后已经前进、推送失败或上下文不一致时立即保存 `blocked` checkpoint;不得盲目重试未知的状态变更。
|
||||
- supervisor 进程退出后,再次执行同一命令即可从当前队列恢复;不复制旧日志、不重放未知命令。
|
||||
- supervisor 只负责有效 handoff 的连续执行,不能绕过真实阻塞、治理门禁、源码编译失败或 provenance 规则。
|
||||
- guardian 状态写入 `tests/golden/WBV2/continuous-guardian.json`,详细输出写入 `continuous-guardian.log`;两个锁文件均防止重复实例。
|
||||
- runner 的 push 流程最多退避重试 6 次,并在每次提交后执行 `git ls-remote origin refs/heads/main` 对比本地 HEAD;未确认远端提交不得进入下一任务。
|
||||
- supervisor 意外退出时,guardian 仅当 checkpoint 为 `handed_off` 且当前队列与 checkpoint.after 完全一致时自动拉起;这是唯一可证明不会重放未知写操作的恢复路径。
|
||||
- 子进程失败、队列未前进、队列在失败后已经前进、推送无法确认或上下文不一致时保存 `blocked` checkpoint;不得盲目重试未知的状态变更。
|
||||
- guardian 或 supervisor 进程退出后,再次执行同一 guardian 命令即可从当前队列恢复;不复制旧日志、不重放未知命令。
|
||||
- guardian 只负责有效 handoff 和可证明的进程恢复,不能绕过真实阻塞、治理门禁、源码编译失败或 provenance 规则。
|
||||
|
||||
## Copyable Resume Prompt
|
||||
|
||||
@@ -113,14 +117,16 @@ and (when authorized) repository push:
|
||||
6. 原始 P0 inventory、6900 总数、6618 legacy pending 和历史 provenance 始终只读。effective completed/pending 只由后续 hash-bound 验证 overlay 计算;实现完成不等于条目完成。
|
||||
7. 长输出写入 `tests/golden/<task>/command.log`;交互只报告退出码、短尾部、构件路径和 SHA-256。
|
||||
8. 任务成功后运行 `node tools/web/check-task-context.mjs --task <task-id> --write`、`node tools/web/check-context-governance.mjs`、`node tools/web/check-v2-task-index.mjs`、`npm --prefix web run typecheck`、`npm --prefix web run lint` 和 `git diff --check`;完整生产验证门禁统一在验证波次执行。
|
||||
9. 所有实现门禁通过后自动提交并推送 `origin/main`,从新 parent manifest 重新读取 `nextTask` 继续。凭据不得写入文件、URL、日志或提示词。
|
||||
9. 所有实现门禁通过后自动提交并推送 `origin/main`;必须再次用 `git ls-remote` 确认远端 `main` 与本地 `HEAD` 完全相同,确认完成后才从新 parent manifest 读取 `nextTask` 继续。凭据不得写入文件、URL、日志或提示词。
|
||||
10. 任一命令失败、输入缺失、队列不一致、环境不可用或状态不明时,保持 `in_progress`/`blocked`,不得推进指针、伪造完成或重复未知结果的状态改变命令。
|
||||
11. 每轮结束报告实现切片、当前任务、`nextTask`、最小检查退出码、原始 inventory、effective inventory 是否保持不变、提交/推送状态和剩余真实阻塞。
|
||||
12. CONTINUOUS_LOOP_REQUIRED:成功交接后立即开始下一轮;不得在成功提交、推送或生成新 manifest 后等待用户确认、只写计划或只汇报状态。每轮必须重新运行 `print-task-context` 和 `check-task-context`,使用新 parent manifest 的 `nextTask`,不沿用上一轮上下文。
|
||||
13. 可使用仓库内 unattended runner 保持进程级连续执行:`node tools/web/run-wbv2-continuous.mjs --max-iterations 0 --commit --push`。它只执行任务卡声明的入口;`0` 表示不限轮数,`--max-iterations N` 用于受控运行,`--commit --push` 才允许自动提交/推送。任何入口失败、退出 0 但队列未前进、治理失败或 push 失败都立即停止并保留 `tests/golden/<task>/continuous-runner.log`,不得重试未知结果。
|
||||
13. 可使用仓库内 guardian 保持进程级连续执行:`node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push`。它托管 supervisor,`0` 表示不限轮数,`--max-iterations N` 用于受控运行,`--commit --push` 才允许自动提交/推送。push 网络抖动由 runner 自动退避重试并确认远端 HEAD;任何入口失败、退出 0 但队列未前进、治理失败、远端无法确认或状态不明都保留 checkpoint,guardian 只自动恢复可证明安全的 handoff。
|
||||
14. 6900_IMPLEMENTATION_IRON_RULE:只按 capability family 和唯一行为批次实现全部 Blender 功能;不启动 Chromium/E2E/receipt/parity/save-reopen/determinism 验证,不写 completion overlay,不把维护 smoke 当作能力完成。若任务卡与本铁律冲突,以源码实现和本铁律为准,并将验证留到后续独立验证波次。
|
||||
15. UNIFIED_WASM_JSON_ABI_IRON_RULE:Worker 与 WASM 的新调用统一使用单一 C ABI `web_engine_call_json(handle, requestJson, resultOut)`;请求必须是 `{"cmd":"...","params":{...}}`,返回必须是 `{"ok":true,"cmd":"...","result":...}` 或包含 `code/error` 的失败 JSON,再由 Worker 解析为领域结果。`applyCommand`、快照、增量、撤销/重做、Depsgraph 等状态路径不得新增独立公开 WASM 函数;open/append/save/geometry 等仍未迁移的二进制专用 ABI 仅作为兼容边界保留,并不得扩散到新的状态调用。
|
||||
16. PROCESS_CONTINUITY_IRON_RULE:要实现跨会话不间断执行,必须使用 `run-wbv2-supervisor.mjs --max-iterations 0`;单次聊天执行完成后不得假定代理会在下一轮自动复活。supervisor 必须按任务原子迭代、锁队列、写 checkpoint、成功后立即重读 manifest;失败或状态未知时只能停在精确 checkpoint,不得自动重试或伪造队列前进。
|
||||
16. PROCESS_CONTINUITY_IRON_RULE:要实现跨会话不间断执行,必须使用 `run-wbv2-guardian.mjs --max-iterations 0` 托管 `run-wbv2-supervisor.mjs`;单次聊天执行完成后不得假定代理会在下一轮自动复活。guardian/supervisor 必须按任务原子迭代、锁队列、写 checkpoint、成功后立即重读 manifest;失败或状态未知时只能停在精确 checkpoint,不得自动重试或伪造队列前进。
|
||||
17. PUSH_COMPLETION_IRON_RULE:任何任务只有在 commit 成功、`git push origin main` 成功且 `git ls-remote --refs origin refs/heads/main` 返回的远端 HEAD 与本地 `git rev-parse HEAD` 相同后才算完成;确认前不得读取下一任务。网络失败最多按退避策略重试,状态未知时不得重放任务写入。
|
||||
18. GUARDIAN_CONTINUITY_IRON_RULE:无人值守运行必须由 `run-wbv2-guardian.mjs` 托管 supervisor。guardian 对可证明的 `handed_off` checkpoint 自动恢复,对 `running`、`started`、`blocked`、指针漂移、远端提交不明或任务闭合状态保持精确停止;“不停歇”仅适用于安全可恢复的进程/网络故障,不能解释为绕过真实阻塞。
|
||||
```
|
||||
|
||||
## Resume Command
|
||||
@@ -150,8 +156,10 @@ review, or diagnosis do not activate this procedure.
|
||||
|
||||
There is no intentional idle step between valid handoffs. Each iteration still
|
||||
has exactly one numbered task and must not carry the previous task's full
|
||||
transcript, logs, or source context. A continuous run may contain many such
|
||||
iterations, but it must stop at the first failed gate or explicit stop request.
|
||||
transcript, logs, or source context. A guardian run may contain many such
|
||||
iterations and automatically recovers process exits after a proven handoff;
|
||||
it stops only at a real gate failure, unknown state, queue closure, or explicit
|
||||
stop request.
|
||||
|
||||
## Continuous Loop Contract
|
||||
|
||||
@@ -188,12 +196,14 @@ not completion overlays.
|
||||
For process-level unattended execution, run:
|
||||
|
||||
```bash
|
||||
node tools/web/run-wbv2-continuous.mjs --max-iterations 0 --commit --push
|
||||
node tools/web/run-wbv2-guardian.mjs --max-iterations 0 --commit --push
|
||||
```
|
||||
|
||||
The runner is intentionally fail-closed. It derives the command from the
|
||||
current task card, requires the queue to advance after a successful exit, and
|
||||
writes one bounded log per task. It never invents a task, retries an unknown
|
||||
writes one bounded log per task. Push completion is accepted only after the
|
||||
remote `origin/main` HEAD equals the local commit. The guardian retries only a
|
||||
proven `handed_off` process exit; it never invents a task, retries an unknown
|
||||
state-changing result, or turns implementation output into parity completion.
|
||||
|
||||
## Activation Contract
|
||||
@@ -224,9 +234,12 @@ hashes, and governance checks agree. A task that fails, is cancelled, exceeds
|
||||
budget, lacks its environment, or has hash drift remains `in_progress` or
|
||||
`blocked`; the next invocation resumes that same task instead of advancing.
|
||||
|
||||
## Stop Conditions
|
||||
## Stop Conditions And Recovery
|
||||
|
||||
Continuous mode must stop immediately when any of these conditions occurs:
|
||||
The unattended guardian does not stop for a transient process or push transport
|
||||
failure. The runner retries push/remote-HEAD confirmation with bounded
|
||||
exponential backoff, and the guardian restarts only after a proven handoff.
|
||||
Continuous mode stops only when an unsafe or irreversible condition remains:
|
||||
|
||||
- a focused command, exit criterion, governance check, or hash check fails;
|
||||
- the queue, parent manifest, task card, task index, or `nextTask` pointer is
|
||||
@@ -234,14 +247,16 @@ Continuous mode must stop immediately when any of these conditions occurs:
|
||||
- the environment is missing, the task is cancelled, or a context/size budget
|
||||
would be exceeded;
|
||||
- a stream disconnect, remote compact error, or other transport error makes the
|
||||
result of a state-changing command unknown;
|
||||
result of a state-changing command unknown after read-only reconciliation;
|
||||
- the queue has no next task or reaches its declared closure gate;
|
||||
- the user explicitly asks to stop or pause.
|
||||
|
||||
On a stop, preserve the current checkpoint, do not start another task, and
|
||||
report the exact task, state, failed gate or stop reason, and evidence paths.
|
||||
Only a successful handoff starts another iteration. There is no retry loop for
|
||||
an unknown or failed result.
|
||||
On an unsafe stop, preserve the current checkpoint, do not start another task,
|
||||
and report the exact task, state, failed gate or stop reason, and evidence paths.
|
||||
Only a successful handoff starts another iteration. A process exit is not by
|
||||
itself a reason to stop: the guardian re-enters only from a matching
|
||||
`handed_off` checkpoint; `running`/`started` or mismatched pointers remain
|
||||
blocked to prevent duplicate writes.
|
||||
|
||||
For every task, run this exact order:
|
||||
|
||||
@@ -285,9 +300,10 @@ to the newly completed manifest pointer.
|
||||
## Transport/Compaction Recovery
|
||||
|
||||
`stream disconnected`, `error decoding response body`, or a remote compact
|
||||
failure means that the request outcome is unknown. It is not evidence that the
|
||||
task failed or succeeded. Do not blindly repeat a generator, build, save, or
|
||||
queue-writing command.
|
||||
failure means that the request outcome is unknown. The process-level guardian
|
||||
continues independently when its checkpoint is valid; a chat client must first
|
||||
perform read-only reconciliation. Do not blindly repeat a generator, build,
|
||||
save, or queue-writing command.
|
||||
|
||||
Recover with read-only checks first:
|
||||
|
||||
@@ -301,9 +317,11 @@ Then compare the current task's manifest, status, task-context, and artifact
|
||||
hashes. Continue from the first missing handoff step; if every criterion is
|
||||
already recorded, run only the missing governance/check command. The failed
|
||||
request's outcome is unknown, so do not claim success from the error message.
|
||||
Resume the request at most once. If the resumed request disconnects again,
|
||||
leave the task `in_progress`/`blocked`, preserve all files, and report the exact
|
||||
command and checkpoint state rather than starting a retry loop.
|
||||
Resume the request at most once in a chat session. For unattended execution,
|
||||
push transport retries are handled by the runner and proven-handoff process
|
||||
restarts by the guardian; if reconciliation cannot prove safety, leave the
|
||||
task `in_progress`/`blocked`, preserve all files, and report the exact command
|
||||
and checkpoint state rather than starting an unsafe retry loop.
|
||||
|
||||
If the client supplies a session id, `codex resume <session-id>` may be used,
|
||||
but the resumed turn must still reread the queue and print a fresh context. Do
|
||||
|
||||
@@ -11,8 +11,18 @@ const maxIterations = maxArg >= 0 ? Number(process.argv[maxArg + 1]) : 0;
|
||||
if (!Number.isSafeInteger(maxIterations) || maxIterations < 0) throw new Error("--max-iterations must be a non-negative integer; 0 means unlimited");
|
||||
const commitEnabled = process.argv.includes("--commit");
|
||||
const pushEnabled = process.argv.includes("--push");
|
||||
if (pushEnabled && !commitEnabled) throw new Error("--push requires --commit");
|
||||
const pushRetryArg = process.argv.indexOf("--push-retries");
|
||||
const pushRetries = pushRetryArg >= 0 ? Number(process.argv[pushRetryArg + 1]) : 6;
|
||||
if (!Number.isSafeInteger(pushRetries) || pushRetries < 0) throw new Error("--push-retries must be a non-negative integer");
|
||||
const shellQuote = (value) => `'${value.replaceAll("'", "'\\''")}'`;
|
||||
|
||||
function waitMs(milliseconds) {
|
||||
if (milliseconds <= 0) return;
|
||||
const wait = new Int32Array(new SharedArrayBuffer(4));
|
||||
Atomics.wait(wait, 0, 0, milliseconds);
|
||||
}
|
||||
|
||||
function command(command, logPath) {
|
||||
const result = spawnSync("bash", ["-lc", command], { cwd: root, encoding: "utf8", maxBuffer: 16 * 1024 * 1024 });
|
||||
const output = `${result.stdout ?? ""}${result.stderr ?? ""}`;
|
||||
@@ -78,8 +88,23 @@ while (maxIterations === 0 || iterations < maxIterations) {
|
||||
const commitResult = command(`git diff --cached --quiet || git commit -m "chore(web): continuous handoff ${before.task}"`, `tests/golden/${before.task}/continuous-commit-result.log`);
|
||||
if (commitResult.code !== 0) throw new Error(`${before.task} commit failed; log=tests/golden/${before.task}/continuous-commit-result.log`);
|
||||
if (pushEnabled) {
|
||||
const push = command("git push origin main", `tests/golden/${before.task}/continuous-push.log`);
|
||||
if (push.code !== 0) throw new Error(`${before.task} push failed; log=tests/golden/${before.task}/continuous-push.log`);
|
||||
let pushed = false;
|
||||
for (let attempt = 0; attempt <= pushRetries; attempt += 1) {
|
||||
const pushLog = attempt === 0
|
||||
? `tests/golden/${before.task}/continuous-push.log`
|
||||
: `tests/golden/${before.task}/continuous-push-attempt-${attempt + 1}.log`;
|
||||
const push = command("git push origin main", pushLog);
|
||||
const pushVerification = command(
|
||||
"local_head=$(git rev-parse --verify HEAD) && remote_head=$(git ls-remote --refs origin refs/heads/main | awk '$2 == \"refs/heads/main\" { print $1; exit }') && printf 'local_head=%s remote_head=%s\\n' \"$local_head\" \"$remote_head\" && test -n \"$remote_head\" && test \"$local_head\" = \"$remote_head\"",
|
||||
`tests/golden/${before.task}/continuous-push-verify.log`,
|
||||
);
|
||||
if (pushVerification.code === 0) {
|
||||
pushed = true;
|
||||
break;
|
||||
}
|
||||
if (attempt < pushRetries) waitMs(Math.min(30000, 1000 * 2 ** attempt));
|
||||
}
|
||||
if (!pushed) throw new Error(`${before.task} remote push verification failed after ${pushRetries + 1} attempts; log=tests/golden/${before.task}/continuous-push-verify.log`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
189
tools/web/run-wbv2-guardian.mjs
Normal file
189
tools/web/run-wbv2-guardian.mjs
Normal file
@@ -0,0 +1,189 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { spawn } from "node:child_process";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../..");
|
||||
const abs = (file) => path.join(root, file);
|
||||
const controlRoot = "tests/golden/WBV2";
|
||||
const logPath = `${controlRoot}/continuous-guardian.log`;
|
||||
const checkpointPath = `${controlRoot}/continuous-guardian.json`;
|
||||
const lockPath = `${controlRoot}/continuous-guardian.lock`;
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
const numericArg = (name, fallback) => {
|
||||
const index = argv.indexOf(name);
|
||||
return Number(index >= 0 ? argv[index + 1] : fallback);
|
||||
};
|
||||
const restartDelayMs = numericArg("--restart-delay-ms", 5000);
|
||||
const maxRestarts = numericArg("--max-restarts", 0);
|
||||
const maxIterations = numericArg("--max-iterations", 0);
|
||||
if (!Number.isSafeInteger(restartDelayMs) || restartDelayMs < 0) throw new Error("--restart-delay-ms must be a non-negative integer");
|
||||
if (!Number.isSafeInteger(maxRestarts) || maxRestarts < 0) throw new Error("--max-restarts must be a non-negative integer; 0 means unlimited safe restarts");
|
||||
if (!Number.isSafeInteger(maxIterations) || maxIterations < 0) throw new Error("--max-iterations must be a non-negative integer; 0 means unlimited");
|
||||
if (argv.includes("--push") && !argv.includes("--commit")) throw new Error("--push requires --commit");
|
||||
|
||||
const supervisorArgs = [];
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
if (argv[index] === "--restart-delay-ms" || argv[index] === "--max-restarts") {
|
||||
index += 1;
|
||||
continue;
|
||||
}
|
||||
supervisorArgs.push(argv[index]);
|
||||
}
|
||||
const supervisorCommand = ["tools/web/run-wbv2-supervisor.mjs", ...supervisorArgs].join(" ");
|
||||
const supervisorNodeArgs = ["tools/web/run-wbv2-supervisor.mjs", ...supervisorArgs];
|
||||
|
||||
fs.mkdirSync(abs(controlRoot), { recursive: true });
|
||||
let lockHandle;
|
||||
const acquireLock = () => {
|
||||
try {
|
||||
return fs.openSync(abs(lockPath), "wx");
|
||||
}
|
||||
catch (error) {
|
||||
if (error?.code !== "EEXIST") throw error;
|
||||
try {
|
||||
const lock = JSON.parse(fs.readFileSync(abs(lockPath), "utf8"));
|
||||
if (Number.isSafeInteger(lock.pid)) {
|
||||
try { process.kill(lock.pid, 0); }
|
||||
catch (probeError) {
|
||||
if (probeError?.code === "ESRCH") {
|
||||
fs.unlinkSync(abs(lockPath));
|
||||
return fs.openSync(abs(lockPath), "wx");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
catch {
|
||||
// An unreadable lock is treated as owned to avoid concurrent guardians.
|
||||
}
|
||||
throw new Error(`another WBV2 guardian appears to be running; lock=${lockPath}`);
|
||||
}
|
||||
};
|
||||
lockHandle = acquireLock();
|
||||
fs.writeFileSync(lockHandle, `${JSON.stringify({ pid: process.pid, startedAt: new Date().toISOString() })}\n`);
|
||||
|
||||
let released = false;
|
||||
const releaseLock = () => {
|
||||
if (released || lockHandle === undefined) return;
|
||||
released = true;
|
||||
fs.closeSync(lockHandle);
|
||||
try { fs.unlinkSync(abs(lockPath)); }
|
||||
catch (error) { if (error?.code !== "ENOENT") throw error; }
|
||||
};
|
||||
const appendLog = (text) => fs.appendFileSync(abs(logPath), `[${new Date().toISOString()}] ${text}\n`);
|
||||
const writeCheckpoint = (payload) => fs.writeFileSync(abs(checkpointPath), `${JSON.stringify({ schemaVersion: 1, pid: process.pid, updatedAt: new Date().toISOString(), ...payload }, null, 2)}\n`);
|
||||
const readJson = (file) => {
|
||||
try { return JSON.parse(fs.readFileSync(abs(file), "utf8")); }
|
||||
catch { return null; }
|
||||
};
|
||||
const currentTask = () => {
|
||||
const queue = fs.readFileSync(abs("docs/EXECUTION_QUEUE.md"), "utf8");
|
||||
const task = queue.match(/\| 当前任务 \| `(WBV2-P\d+-\d+)` \|/u)?.[1];
|
||||
const parentManifest = queue.match(/\| parent manifest \| `([^`]+)` \|/u)?.[1];
|
||||
if (!task || !parentManifest) throw new Error("queue pointer is incomplete");
|
||||
return { task, parentManifest };
|
||||
};
|
||||
const samePointer = (left, right) => left?.task === right?.task && left?.parentManifest === right?.parentManifest;
|
||||
const sleep = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds));
|
||||
|
||||
let child;
|
||||
let stopRequested = false;
|
||||
let stopSignal;
|
||||
const signalChild = (signal) => {
|
||||
if (!child?.pid) return;
|
||||
try { process.kill(-child.pid, signal); }
|
||||
catch (error) {
|
||||
if (error?.code !== "ESRCH") {
|
||||
try { child.kill(signal); }
|
||||
catch { /* The exit handler will record the final state. */ }
|
||||
}
|
||||
}
|
||||
};
|
||||
const requestStop = (signal) => {
|
||||
stopRequested = true;
|
||||
stopSignal = signal;
|
||||
appendLog(`stop-requested signal=${signal}`);
|
||||
signalChild(signal);
|
||||
};
|
||||
process.once("SIGINT", () => requestStop("SIGINT"));
|
||||
process.once("SIGTERM", () => requestStop("SIGTERM"));
|
||||
|
||||
const runSupervisor = () => new Promise((resolve) => {
|
||||
child = spawn(process.execPath, supervisorNodeArgs, {
|
||||
cwd: root,
|
||||
detached: true,
|
||||
env: process.env,
|
||||
stdio: "inherit",
|
||||
});
|
||||
child.once("error", (error) => resolve({ code: 1, signal: null, error: error.message }));
|
||||
child.once("exit", (code, signal) => {
|
||||
child = undefined;
|
||||
resolve({ code: code ?? 1, signal });
|
||||
});
|
||||
});
|
||||
|
||||
function safeRestart(checkpoint, pointer) {
|
||||
if (!checkpoint) return { ok: false, reason: "supervisor_checkpoint_missing" };
|
||||
if (checkpoint.state === "blocked") return { ok: false, reason: checkpoint.reason ?? "supervisor_blocked" };
|
||||
if (checkpoint.state === "stopped") return { ok: false, reason: checkpoint.reason ?? "supervisor_stopped" };
|
||||
if (checkpoint.state !== "handed_off") return { ok: false, reason: `supervisor_state_${checkpoint.state ?? "unknown"}` };
|
||||
if (!samePointer(checkpoint.after, pointer)) return { ok: false, reason: "handoff_pointer_mismatch" };
|
||||
return { ok: true };
|
||||
}
|
||||
|
||||
let guardianExit = 0;
|
||||
let restartCount = 0;
|
||||
try {
|
||||
appendLog(`started pid=${process.pid} supervisor=${supervisorCommand} maxRestarts=${maxRestarts} restartDelayMs=${restartDelayMs}`);
|
||||
writeCheckpoint({ state: "started", restartCount, supervisor: supervisorCommand, current: currentTask() });
|
||||
while (true) {
|
||||
if (stopRequested) {
|
||||
writeCheckpoint({ state: "stopped", restartCount, reason: stopSignal, current: currentTask() });
|
||||
guardianExit = stopSignal === "SIGINT" ? 130 : 143;
|
||||
break;
|
||||
}
|
||||
const before = currentTask();
|
||||
writeCheckpoint({ state: "running", restartCount, before, supervisor: supervisorCommand });
|
||||
const result = await runSupervisor();
|
||||
const checkpoint = readJson(checkpointPath);
|
||||
const after = currentTask();
|
||||
appendLog(`supervisor-exit code=${result.code} signal=${result.signal ?? "none"} task=${before.task} queue=${after.task} state=${checkpoint?.state ?? "missing"}`);
|
||||
if (stopRequested) {
|
||||
writeCheckpoint({ state: "stopped", restartCount, reason: stopSignal, before, after, supervisorExit: result.code });
|
||||
guardianExit = stopSignal === "SIGINT" ? 130 : 143;
|
||||
break;
|
||||
}
|
||||
if (checkpoint?.state === "complete") {
|
||||
writeCheckpoint({ state: "complete", restartCount, current: after, supervisorExit: result.code });
|
||||
break;
|
||||
}
|
||||
const decision = safeRestart(checkpoint, after);
|
||||
if (!decision.ok) {
|
||||
writeCheckpoint({ state: "blocked", restartCount, before, after, reason: decision.reason, supervisorExit: result.code });
|
||||
appendLog(`blocked reason=${decision.reason}; no automatic replay`);
|
||||
guardianExit = 2;
|
||||
break;
|
||||
}
|
||||
restartCount += 1;
|
||||
if (maxRestarts > 0 && restartCount > maxRestarts) {
|
||||
writeCheckpoint({ state: "blocked", restartCount, before, after, reason: "max_safe_restarts_exceeded", supervisorExit: result.code });
|
||||
appendLog("blocked reason=max_safe_restarts_exceeded");
|
||||
guardianExit = 3;
|
||||
break;
|
||||
}
|
||||
writeCheckpoint({ state: "restarting", restartCount, after, reason: "safe_handoff_checkpoint" });
|
||||
appendLog(`restarting count=${restartCount} after=${after.task}`);
|
||||
await sleep(restartDelayMs);
|
||||
}
|
||||
}
|
||||
catch (error) {
|
||||
guardianExit = 4;
|
||||
writeCheckpoint({ state: "blocked", restartCount, reason: "guardian_error", error: error?.message ?? String(error) });
|
||||
appendLog(`blocked reason=guardian_error error=${error?.message ?? String(error)}`);
|
||||
}
|
||||
finally {
|
||||
signalChild("SIGTERM");
|
||||
releaseLock();
|
||||
}
|
||||
process.exitCode = guardianExit;
|
||||
@@ -16,10 +16,12 @@ const valueArg = (name, fallback) => {
|
||||
};
|
||||
const maxIterations = Number(valueArg("--max-iterations", "0"));
|
||||
const pollMs = Number(valueArg("--poll-ms", "1000"));
|
||||
const pushRetries = Number(valueArg("--push-retries", "6"));
|
||||
const commitEnabled = process.argv.includes("--commit");
|
||||
const pushEnabled = process.argv.includes("--push");
|
||||
if (!Number.isSafeInteger(maxIterations) || maxIterations < 0) throw new Error("--max-iterations must be a non-negative integer; 0 means unlimited");
|
||||
if (!Number.isSafeInteger(pollMs) || pollMs < 0) throw new Error("--poll-ms must be a non-negative integer");
|
||||
if (!Number.isSafeInteger(pushRetries) || pushRetries < 0) throw new Error("--push-retries must be a non-negative integer");
|
||||
if (pushEnabled && !commitEnabled) throw new Error("--push requires --commit");
|
||||
|
||||
fs.mkdirSync(abs(supervisorRoot), { recursive: true });
|
||||
@@ -59,19 +61,34 @@ const releaseLock = () => {
|
||||
try { fs.unlinkSync(abs(lockPath)); }
|
||||
catch (error) { if (error?.code !== "ENOENT") throw error; }
|
||||
};
|
||||
let stopRequested = false;
|
||||
let stopSignal;
|
||||
let iteration = 0;
|
||||
let fatalHandled = false;
|
||||
const stopFromSignal = (signal) => {
|
||||
try {
|
||||
writeCheckpoint({ state: "stopped", iteration: 0, reason: signal });
|
||||
appendLog(`stopped signal=${signal}`);
|
||||
releaseLock();
|
||||
}
|
||||
finally {
|
||||
process.exitCode = signal === "SIGINT" ? 130 : 143;
|
||||
}
|
||||
stopRequested = true;
|
||||
stopSignal = signal;
|
||||
appendLog(`stop-requested signal=${signal}`);
|
||||
process.exitCode = signal === "SIGINT" ? 130 : 143;
|
||||
};
|
||||
process.once("SIGINT", () => stopFromSignal("SIGINT"));
|
||||
process.once("SIGTERM", () => stopFromSignal("SIGTERM"));
|
||||
|
||||
const stopOnFatal = (error, source) => {
|
||||
if (fatalHandled) return;
|
||||
fatalHandled = true;
|
||||
try {
|
||||
writeCheckpoint({ state: "blocked", iteration, reason: source, error: error?.message ?? String(error) });
|
||||
appendLog(`blocked reason=${source} error=${error?.message ?? String(error)}`);
|
||||
}
|
||||
finally {
|
||||
releaseLock();
|
||||
process.exit(1);
|
||||
}
|
||||
};
|
||||
process.once("uncaughtException", (error) => stopOnFatal(error, "uncaught_exception"));
|
||||
process.once("unhandledRejection", (error) => stopOnFatal(error, "unhandled_rejection"));
|
||||
|
||||
if (lockHandle === undefined) {
|
||||
throw new Error(`unable to acquire supervisor lock=${lockPath}`);
|
||||
}
|
||||
@@ -96,14 +113,18 @@ const currentTask = () => {
|
||||
const childArgs = ["tools/web/run-wbv2-continuous.mjs", "--max-iterations", "1"];
|
||||
if (commitEnabled) childArgs.push("--commit");
|
||||
if (pushEnabled) childArgs.push("--push");
|
||||
if (pushEnabled) childArgs.push("--push-retries", String(pushRetries));
|
||||
|
||||
appendLog(`started pid=${process.pid} maxIterations=${maxIterations} child=${childArgs.join(" ")}`);
|
||||
writeCheckpoint({ state: "started", iteration: 0, current: currentTask(), child: childArgs.join(" ") });
|
||||
|
||||
let iteration = 0;
|
||||
let exitCode = 0;
|
||||
try {
|
||||
while (maxIterations === 0 || iteration < maxIterations) {
|
||||
if (stopRequested) {
|
||||
writeCheckpoint({ state: "stopped", iteration, current: currentTask(), reason: stopSignal });
|
||||
break;
|
||||
}
|
||||
const before = currentTask();
|
||||
const print = run(process.execPath, ["tools/web/print-task-context.mjs"]);
|
||||
const check = run(process.execPath, ["tools/web/check-task-context.mjs"]);
|
||||
@@ -117,6 +138,10 @@ try {
|
||||
const child = run(process.execPath, childArgs);
|
||||
appendLog(`child task=${before.task} exit=${child.code}\n${child.output}`);
|
||||
const after = currentTask();
|
||||
if (stopRequested) {
|
||||
writeCheckpoint({ state: "stopped", iteration, before, after, reason: stopSignal, childExit: child.code });
|
||||
break;
|
||||
}
|
||||
if (child.code !== 0) {
|
||||
const reason = after.task === before.task ? "child_failed_same_task" : "child_failed_after_queue_advance";
|
||||
writeCheckpoint({ state: "blocked", iteration: iteration + 1, before, after, reason, childExit: child.code });
|
||||
@@ -136,6 +161,9 @@ try {
|
||||
Atomics.wait(wait, 0, 0, pollMs);
|
||||
}
|
||||
}
|
||||
if (stopRequested) {
|
||||
exitCode = stopSignal === "SIGINT" ? 130 : 143;
|
||||
}
|
||||
if (exitCode === 0) {
|
||||
const current = currentTask();
|
||||
writeCheckpoint({ state: maxIterations === 0 ? "running" : "complete", iteration, current });
|
||||
|
||||
Reference in New Issue
Block a user