Merge pull request #3211 from deepseek-harness/ci/gate-fail-fast
ci: fail fast at the first blocking gate failure
This commit is contained in:
Chinesezjc
GitHub
commit
456dcdd8fb
10 changed files with 1084 additions and 48 deletions
|
|
@ -2,5 +2,5 @@
|
|||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-06-parallel-pre-push-gates.md
|
||||
2026-07-06-parallel-pre-push-gates.md: 22d69478f0fe664b91c4ada2c5c97e7c61ee7deb
|
||||
2026-07-06-parallel-pre-push-gates.zh.md: 98de527688399b8f6c09e91f55916361bf79d12d
|
||||
2026-07-06-parallel-pre-push-gates.md: 54fb01f03de1d0d198e373d960e9bd68b8687d60
|
||||
2026-07-06-parallel-pre-push-gates.zh.md: d7a949af649d3cf83da91358015f9196f71bc459
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ Status: implemented
|
|||
|
||||
English | [中文](2026-07-06-parallel-pre-push-gates.zh.md)
|
||||
|
||||
The local-hook portion of this record is superseded by [Fast local Git hooks](2026-07-22-fast-local-git-hooks.md). The bounded gate scheduler and package-level `publint` parallelism remain in force for CI, `doc-sync`, and explicit local commands.
|
||||
The local-hook portion of this record is superseded by [Fast local Git hooks](2026-07-22-fast-local-git-hooks.md). The bounded gate scheduler and package-level `publint` parallelism remain in force for CI, `doc-sync`, and explicit local commands. The scheduler's fail-fast option is recorded in [Gate-runner fail-fast](2026-08-27-gate-runner-fail-fast.md).
|
||||
|
||||
## Problem
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ Status: implemented
|
|||
|
||||
[English](2026-07-06-parallel-pre-push-gates.md) | 中文
|
||||
|
||||
本记录中的本地钩子部分已由[快速本地 Git 钩子](2026-07-22-fast-local-git-hooks.zh.md) 取代。有界门禁调度器和包级 `publint` 并行机制仍用于 CI、`doc-sync` 和显式本地命令。
|
||||
本记录中的本地钩子部分已由[快速本地 Git 钩子](2026-07-22-fast-local-git-hooks.zh.md) 取代。有界门禁调度器和包级 `publint` 并行机制仍用于 CI、`doc-sync` 和显式本地命令。调度器的快速失败选项记录在[门禁运行器快速失败](2026-08-27-gate-runner-fail-fast.zh.md)。
|
||||
|
||||
## 问题
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,6 @@
|
|||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-08-27-gate-runner-fail-fast.md
|
||||
2026-08-27-gate-runner-fail-fast.md: b11e3336aad01d2d4bf57c132e1aeaddfac5c258
|
||||
2026-08-27-gate-runner-fail-fast.zh.md: 93e1b0314289afa17e89afea19c8b8c0563362f4
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
# Agent Note: Gate-runner fail-fast
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-08-27-gate-runner-fail-fast.zh.md)
|
||||
|
||||
[Parallel pre-push gates](2026-07-06-parallel-pre-push-gates.md) owns the bounded gate scheduler in `scripts/run-gates.ts`; this note adds one scheduling option to that scheduler.
|
||||
|
||||
## Problem
|
||||
|
||||
The gate scheduler in `scripts/run-gates.ts` runs every independent gate in an aggregate to completion and reports `run-gates: N passed, M failed`. A gate failure does not stop the remaining gates; only gates whose `needs` dependency failed are skipped. On an aggregate that is already red, the remaining gates keep consuming runner time and produce evidence that cannot change the verdict. The largest single cost is the instrumented coverage run in the `ci-coverage` aggregate, which has taken about 27 minutes; in `ci-consumers`, the Node compatibility smoke runs independently of the build, so it keeps running after a build failure that already settles the verdict.
|
||||
|
||||
GitHub Actions provides no native cross-job cancellation: `fail-fast` applies only inside a matrix, and the `all checks passed` aggregate settles only after every needed job finishes, so it cannot cancel siblings early. The only in-repository lever is the gate scheduler itself.
|
||||
|
||||
## Decision
|
||||
|
||||
`run-gates.ts` accepts a fail-fast scheduling option. When enabled, the first blocking gate failure (a gate whose `allowFailure` is not true) aborts the aggregate: the shared `AbortSignal` terminates every running gate's process tree, and every not-yet-run gate is recorded as `skipped` with the error `aborted by fail-fast: <label> failed` (or `aborted by fail-fast: host interruption` when the host signal aborted the run). The exit status remains 1, including when a killed child traps the signal and exits zero: such a result carries the abort mark and is recorded `skipped`, never passed. A gate that settled before the abort took effect keeps its real result, so the summary stays truthful about what produced evidence.
|
||||
|
||||
Termination covers the whole tree, not just the direct pnpm wrapper: POSIX signals the detached child's process group (`kill(-pid, SIGTERM)`, escalating unconditionally to `SIGKILL` after 5 seconds) and additionally signals every transitive descendant read from the process table, so the detached leaves of a nested run-gates (the `check:node-compat` and `check:ci:lint:contracts-ready` gates inside `ci-consumers`) are killed without relying on the inner scheduler's own escalation. The descendant list is primed at spawn and refreshed every 5 seconds while the child runs, each tick merging the fresh enumeration into the live-filtered cache (the enumeration is asynchronous — a slow WMI/CIM call is bounded by its own 10-second timeout and never blocks the gate's output draining or exit handling; an enumeration still in flight when the gate settles is cancelled rather than left holding stdio handles, and a snapshot that settles after the child exited is still merged, because the child may be gone while a grandchild keeps `close` pending — exactly when the abort needs the list) so a descendant reparented by an exited intermediate stays tracked across ticks; at abort the fresh enumeration is merged into the same list, then re-signalled on the escalation, because the group kill reaps the direct child and reparents its detached descendants, making them unreachable by parent id afterwards; settlement waits until the group and the captured descendants are gone. On the abort path only, a bounded pipe-drain timer force-closes the stdio streams 10 seconds after the abort, so a descendant holding the write ends (uninterruptible I/O included) cannot keep `close` pending to the job timeout; ordinary runs keep waiting rather than report passed over a live leak. Windows runs `taskkill /PID <pid> /T /F` immediately, because a taskkill without `/F` does not terminate console processes, which is what gate commands are; the same process-table enumeration as POSIX supplies a descendant list there, and each captured descendant is also taskkilled, because a `taskkill /T` rooted at a pid that already exited finds nothing. Windows never reparents, so an exited root's descendants keep it as their parent and remain reachable through the table; the same sampler cadence as POSIX keeps the cache crossing a vanished intermediate's table record (the enumeration is bounded by a 10-second PowerShell timeout so a hung WMI/CIM call cannot stall the abort path). Without this, Windows has no signal forwarding and a wrapper-only kill would orphan the script tree on the shared self-hosted pool. Children are detached into their own POSIX process group only when fail-fast is enabled; ordinary runs keep them in the host group so terminal Ctrl+C still reaches them. Host `SIGINT`/`SIGTERM` on a fail-fast run is forwarded to the abort path, so an interrupted or runner-cancelled run drains and kills its gate trees instead of orphaning them.
|
||||
|
||||
The option is enabled through `DSH_GATE_FAIL_FAST` (accepted values: `1` or unset; anything else fails loud through the existing `flagEnabled` contract) on every run-gates aggregate job in `ci.yml`: the three blocking Linux jobs (`node-24` static, `node-24-coverage`, `node-24-consumers`), the Node compatibility matrix (`node-compat`), and the two native Windows lanes that drive aggregates (`windows-build`, `windows-coverage`). `scripts/ci-workflow.spec.ts` pins the flag on those jobs and pins its absence on `windows-observational`, so removing it fails the CI gate.
|
||||
|
||||
The `windows-observational` lane stays complete: it is `continue-on-error` by design and exists to collect as much Windows-native evidence per run as possible, so the first failure must not truncate the rest. The `windows` Wine lane and `windows-native-tests` run a single script or Vitest command rather than a run-gates aggregate, so the scheduler option does not apply to them. The master serial standby lanes (`serial-linux-selfhosted`, `serial-windows`) and the manual runner benchmarks do not set the flag: they are completeness drills that must execute the full aggregate to prove pool readiness.
|
||||
|
||||
## Consequences
|
||||
|
||||
A red pull-request run ends sooner. The largest saving is in `ci-coverage`: a failing exempt-heavy gate aborts the multi-minute instrumented coverage gate instead of letting it run out.
|
||||
|
||||
The trade-off is diagnostic: one push returns only the first blocking failure instead of the full failure set, so resolving several independent failures may take more push-fix rounds. Killed gates are recorded as `skipped` with the fail-fast error, so the summary line `N passed, M failed, K skipped` remains truthful about what produced evidence and what did not. A gate that ignores `SIGTERM` is force-killed after the 5-second grace. A tree that survives both signals holds the aggregate only while its direct child's stdio stays open; once `close` fires, the group-liveness poll gives up after 8 seconds and the run settles with a loud `gate tree not quiescent` warning instead of reporting a clean tree.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Cross-job cancellation watchdog.** A job that polls sibling conclusions and calls the run-cancel API would stop all lanes on the first failure. It is not native, adds a polling dependency and token surface, and discards the parallel evidence other jobs have already produced. Rejected; fail-fast at the scheduler is orthogonal to the job topology and carries none of that.
|
||||
|
||||
**Consolidating the three Linux jobs into one check.** A single job could fail fast natively, but it would lose the independent runner allocation whose queue-delay overlap is documented in the [independent CI consumer build](2026-07-30-independent-ci-consumer-build.md) note, and it would make the coverage long tail the tail of the whole job. Rejected; fail-fast applies within the existing job split instead.
|
||||
|
||||
**Signaling only the direct child.** The first implementation sent `SIGTERM` to the pnpm wrapper and relied on pnpm forwarding it to the script child. A probe confirmed the forwarding on POSIX. Rejected: Windows has no signal forwarding, and a wrapper-only kill orphans the script tree on the shared self-hosted pool; the tree termination above covers both platforms.
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
# Agent Note: 门禁运行器快速失败
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-08-27-gate-runner-fail-fast.md) | 中文
|
||||
|
||||
[并行推送前门禁](2026-07-06-parallel-pre-push-gates.zh.md)拥有 `scripts/run-gates.ts` 中有界的门禁调度器；本笔记为该调度器新增一个调度选项。
|
||||
|
||||
## 问题
|
||||
|
||||
`scripts/run-gates.ts` 中的门禁调度器会把一个聚合流程里的每个独立门禁都跑完，然后报告 `run-gates: N passed, M failed`。某个门禁失败并不会停止其余门禁；只有 `needs` 依赖失败的门禁会被跳过。当聚合流程已经确定失败时，剩余门禁仍在消耗运行器时间，产出的证据也无法改变结论。最大的单项成本是 `ci-coverage` 聚合流程里的插桩覆盖率运行，实测约 27 分钟；在 `ci-consumers` 中，Node 兼容性冒烟检查与构建相互独立，因此它会在构建失败、结论已定的情况下继续运行。
|
||||
|
||||
GitHub Actions 不提供原生的跨作业取消：`fail-fast` 只作用于矩阵内部，而 `all checks passed` 聚合流程要等所有必需作业结束后才落定，因此它无法提前取消兄弟作业。仓库内唯一可用的杠杆就是门禁调度器本身。
|
||||
|
||||
## 决策
|
||||
|
||||
`run-gates.ts` 接受一个快速失败调度选项。启用后，首个阻塞性门禁失败（`allowFailure` 不为 true 的门禁）即中止整个聚合流程：共享的 `AbortSignal` 终止每个运行中门禁的整个进程树，所有尚未运行的门禁记录为 `skipped`，错误信息为 `aborted by fail-fast: <label> failed`（当宿主信号中止运行时为 `aborted by fail-fast: host interruption`）。退出状态保持为 1——即使被终止的子进程捕获信号并以 0 退出，这类结果也带有中止标记、记为 `skipped`，绝不记为通过。在中止生效前已结算的门禁保留其真实结果，因此汇总行仍如实反映哪些门禁产出了证据。
|
||||
|
||||
终止覆盖整棵进程树，而不只是直接的 pnpm 包装进程：POSIX 向分离子进程所在的进程组发信号（`kill(-pid, SIGTERM)`，5 秒后无条件升级为 `SIGKILL`），并额外按进程表逐个信号所有传递性后代——嵌套 run-gates（`ci-consumers` 里的 `check:node-compat` 与 `check:ci:lint:contracts-ready` 门禁）分离出去的叶子进程因此不依赖内层调度器自己的升级也能被杀掉。后代列表在 spawn 时预填、随后在子进程运行期间每 5 秒刷新（枚举是异步的——慢的 WMI/CIM 调用以自身 10 秒超时封顶，绝不会阻塞门禁的输出排空或退出处理；门禁结算时仍在途的枚举会被取消，而不是留着持有 stdio 句柄；子进程退出后才结算的快照仍会被合并，因为子进程可能已消失而孙进程仍持有 `close` 未触发——这正是中止需要该列表的时刻），每个采样 tick 把新枚举结果并入存活过滤后的缓存，被已退出中间进程 reparent 的后代因此跨 tick 仍被跟踪；中止时把新枚举结果并入同一列表，随后在升级时再次信号：组杀会重收直接子进程并 reparent 其分离后代，之后按父 id 不可达，因此结算要等进程组与已捕获后代都消失。仅在中止路径上，有界 pipe-drain 定时器会在中止后 10 秒强制关闭 stdio 流，因此持有写端的后代进程（包括不可中断 I/O）无法让 `close` 一直挂到作业超时；普通运行保持等待，而不是在有存活泄漏时报告通过。Windows 立即运行 `taskkill /PID <pid> /T /F`，因为不带 `/F` 的 taskkill 无法终止控制台进程，而门禁命令正是控制台进程；与 POSIX 相同的进程表枚举在 Windows 上也提供后代列表，每个捕获的后代同样被 taskkill——因为以已退出 pid 为根执行 `taskkill /T` 找不到任何东西，而 Windows 从不 reparent，已退出根的子孙仍以它为父、通过进程表可达；与 POSIX 相同的采样节奏让缓存跨过已消失中间进程的进程表记录（枚举以 10 秒 PowerShell 超时封顶，挂起的 WMI/CIM 调用不会拖住中止路径）。没有这一步，Windows 上不存在信号转发，只杀包装进程会在共享自托管池上留下孤儿脚本树。只有在启用快速失败时，子进程才会分离到自己的 POSIX 进程组；普通运行保持子进程在宿主进程组内，终端 Ctrl+C 仍能送达它们。快速失败运行上的宿主 `SIGINT`/`SIGTERM` 会转发到中止路径，因此被中断或被运行器取消的运行会排干并杀死自己的门禁进程树，而不是留下孤儿。
|
||||
|
||||
该选项通过 `DSH_GATE_FAIL_FAST` 开启（可接受值：`1` 或未设置；其它值通过既有的 `flagEnabled` 契约响亮失败），作用于 `ci.yml` 中所有由 run-gates 驱动的聚合作业：三个阻塞性 Linux 作业（`node-24` 静态、`node-24-coverage`、`node-24-consumers`）、Node 兼容性矩阵（`node-compat`）以及两个驱动聚合流程的原生 Windows 车道（`windows-build`、`windows-coverage`）。`scripts/ci-workflow.spec.ts` 固定了这些作业上的该标志，并固定 `windows-observational` 上不存在该标志，移除它会令 CI 门禁失败。
|
||||
|
||||
`windows-observational` 车道保持完整执行：它按设计是 `continue-on-error`，存在的意义是每次运行尽量收集 Windows 原生证据，因此首个失败不应截断其余部分。`windows` Wine 车道和 `windows-native-tests` 运行的是单个脚本或 Vitest 命令，不是 run-gates 聚合流程，因此调度选项不适用于它们。master 串行备用车道（`serial-linux-selfhosted`、`serial-windows`）和手动运行器基准测试不设置该标志：它们是完整性演练，必须执行完整聚合流程以证明池的可用性。
|
||||
|
||||
## 结果
|
||||
|
||||
红色拉取请求运行会更早结束。最大节省在 `ci-coverage`：失败的豁免重型门禁会中止多分钟的插桩覆盖率门禁，而不是让它跑完。
|
||||
|
||||
代价在诊断侧：一次推送只返回第一条阻塞性失败，而不是完整失败集，因此解决多个独立失败可能需要更多次推送-修复往返。被终止的门禁记录为带快速失败错误的 `skipped`，因此 `N passed, M failed, K skipped` 汇总行仍然如实反映哪些门禁产出了证据、哪些没有。忽略 `SIGTERM` 的门禁会在 5 秒宽限期后被强制终止。同时扛过两种信号仍存活的进程树，只有在直接子进程的 stdio 保持打开时才会拖住聚合流程；一旦 `close` 触发，进程组存活轮询在 8 秒后放弃，运行以响亮的 `gate tree not quiescent` 警告结算，而不是报告一棵干净的树。
|
||||
|
||||
## 备选方案
|
||||
|
||||
**跨作业取消看门狗。** 一个轮询兄弟作业结论并调用运行取消 API 的作业可以在首个失败时停止所有车道。它不是原生能力，会增加轮询依赖和令牌暴露面，并丢弃其它作业已并行产出的证据。已拒绝；调度器层面的快速失败与作业拓扑正交，不带来上述任何负担。
|
||||
|
||||
**把三个 Linux 作业合并成一个检查。** 单个作业可以原生快速失败，但会失去独立的运行器分配——其排队延迟重叠的收益记录在[独立 CI 消费方构建](2026-07-30-independent-ci-consumer-build.zh.md)笔记中——并且会让覆盖率长尾成为整个作业的尾部。已拒绝；快速失败在既有作业拆分内生效即可。
|
||||
|
||||
**只向直接子进程发信号。** 第一版实现向 pnpm 包装进程发送 `SIGTERM`，依赖 pnpm 把它转发给脚本子进程。探针确认了 POSIX 上的转发。已拒绝：Windows 没有信号转发，只杀包装进程会在共享自托管池上留下孤儿脚本树；上述整树终止覆盖两个平台。
|
||||
19
.github/workflows/ci.yml
vendored
19
.github/workflows/ci.yml
vendored
|
|
@ -46,6 +46,9 @@ jobs:
|
|||
name: node 24 / static
|
||||
env:
|
||||
DSH_GATE_CONCURRENCY: '8'
|
||||
# Stop the aggregate at the first blocking gate failure so a red run
|
||||
# does not keep burning enterprise runner time on the remaining gates.
|
||||
DSH_GATE_FAIL_FAST: '1'
|
||||
steps:
|
||||
# Fetch complete history so the archive gate can read the trusted PR base from a reused shallow checkout.
|
||||
- uses: actions/checkout@v6
|
||||
|
|
@ -102,6 +105,9 @@ jobs:
|
|||
DSH_COVERAGE_MAX_WORKERS: '6'
|
||||
DSH_COVERAGE_PARTITIONS: '4'
|
||||
DSH_GATE_CONCURRENCY: '3'
|
||||
# A gate failure aborts the sibling gate instead of waiting out its
|
||||
# multi-minute instrumented run.
|
||||
DSH_GATE_FAIL_FAST: '1'
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
|
|
@ -164,6 +170,10 @@ jobs:
|
|||
DSH_OXLINT_THREADS: '8'
|
||||
DSH_PUBLINT_CONCURRENCY: '8'
|
||||
DSH_WEB_SNAPSHOT_WORKERS: '6'
|
||||
# A failing gate aborts its running siblings: a failing build stops the
|
||||
# independent Node compatibility smoke, and a failing reader (e.g.
|
||||
# publint) stops the remaining artifact consumers.
|
||||
DSH_GATE_FAIL_FAST: '1'
|
||||
# Failover halves snapshot concurrency for the shared 64-core VM.
|
||||
DSH_SNAPSHOT_MAX_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER_LINUX == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '12' || '32' }}
|
||||
steps:
|
||||
|
|
@ -244,6 +254,9 @@ jobs:
|
|||
env:
|
||||
DSH_GATE_CONCURRENCY: ${{ matrix.gate_concurrency }}
|
||||
DSH_NODE_COMPAT_SKIP_TYPECHECK: '1'
|
||||
# A failed smoke aborts the remaining compatibility gates instead of
|
||||
# letting the build-backed legs run against an already-red aggregate.
|
||||
DSH_GATE_FAIL_FAST: '1'
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
|
|
@ -425,6 +438,9 @@ jobs:
|
|||
|| 'dsh-windows-2025-16core' }}
|
||||
name: windows node 24 / build
|
||||
timeout-minutes: 60
|
||||
env:
|
||||
# A failing build or site aborts the sibling gate on the same runner.
|
||||
DSH_GATE_FAIL_FAST: '1'
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
|
|
@ -471,6 +487,9 @@ jobs:
|
|||
DSH_COVERAGE_PARTITIONS: '4'
|
||||
DSH_COVERAGE_TEST_TIMEOUT_MS: '90000'
|
||||
DSH_GATE_CONCURRENCY: '3'
|
||||
# A gate failure aborts the sibling gate instead of waiting out its
|
||||
# multi-minute instrumented run.
|
||||
DSH_GATE_FAIL_FAST: '1'
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
|
|
|
|||
|
|
@ -67,11 +67,12 @@ describe('CI workflow', () => {
|
|||
|| !isRecord(workflow.jobs['node-24'])
|
||||
|| !isRecord(workflow.jobs['node-24-coverage'])
|
||||
|| !isRecord(workflow.jobs['node-24-consumers'])
|
||||
|| !isRecord(workflow.jobs['node-compat'])
|
||||
|| !isRecord(workflow.jobs['all-checks-passed'])
|
||||
|| !isRecord(masterWorkflow.jobs)
|
||||
|| !isRecord(masterWorkflow.jobs['wine-apt-cache'])
|
||||
|| !isRecord(masterWorkflow.jobs['serial-windows'])) {
|
||||
throw new TypeError('CI workflow must define windows, windows-build, windows-coverage, windows-native-tests, windows-observational, node-24, node-24-coverage, node-24-consumers, and all-checks-passed; ci-master must define wine-apt-cache and serial-windows')
|
||||
throw new TypeError('CI workflow must define windows, windows-build, windows-coverage, windows-native-tests, windows-observational, node-24, node-24-coverage, node-24-consumers, node-compat, and all-checks-passed; ci-master must define wine-apt-cache and serial-windows')
|
||||
}
|
||||
|
||||
const windows = workflow.jobs.windows
|
||||
|
|
@ -84,6 +85,7 @@ describe('CI workflow', () => {
|
|||
const node24 = workflow.jobs['node-24']
|
||||
const node24Coverage = workflow.jobs['node-24-coverage']
|
||||
const node24Consumers = workflow.jobs['node-24-consumers']
|
||||
const nodeCompat = workflow.jobs['node-compat']
|
||||
const aggregate = workflow.jobs['all-checks-passed']
|
||||
if (!Array.isArray(windows.steps) || !Array.isArray(aggregate.needs)) {
|
||||
throw new TypeError('Windows job must define steps and the aggregate must define needs')
|
||||
|
|
@ -222,6 +224,26 @@ describe('CI workflow', () => {
|
|||
expect(aggregate['runs-on']).toContain('DSH_CI_FAILOVER_LINUX')
|
||||
expect(aggregate['runs-on']).not.toContain('DSH_CI_FAILOVER_WINDOWS')
|
||||
expect(aggregate['runs-on']).toContain('vm-backup')
|
||||
|
||||
// The run-gates aggregate lanes stop at the first blocking gate failure so
|
||||
// a red aggregate does not keep burning runner time on the remaining
|
||||
// gates. Removing the flag silently reverts to running every independent
|
||||
// gate to completion.
|
||||
for (const [jobName, job] of [['node-24', node24], ['node-24-coverage', node24Coverage], ['node-24-consumers', node24Consumers], ['node-compat', nodeCompat]] as const) {
|
||||
expect(job.env, `${jobName} must enable fail-fast`).toMatchObject({ DSH_GATE_FAIL_FAST: '1' })
|
||||
}
|
||||
|
||||
// The native Windows lanes with run-gates aggregates fail fast for the
|
||||
// same reason: a failing gate aborts the sibling gate instead of waiting
|
||||
// out the multi-minute instrumented coverage run.
|
||||
expect(windowsBuild.env, 'windows-build must enable fail-fast').toMatchObject({ DSH_GATE_FAIL_FAST: '1' })
|
||||
expect(windowsCoverage.env, 'windows-coverage must enable fail-fast').toMatchObject({ DSH_GATE_FAIL_FAST: '1' })
|
||||
|
||||
// The observational lane stays complete: it is continue-on-error by design
|
||||
// and exists to collect as much Windows-native evidence per run as
|
||||
// possible, so the first failure must not truncate the rest.
|
||||
expect(windowsObservational.env).toBeDefined()
|
||||
expect(windowsObservational.env).not.toMatchObject({ DSH_GATE_FAIL_FAST: '1' })
|
||||
})
|
||||
|
||||
it('gives the Wine Host TypeScript compile the repository heap budget', () => {
|
||||
|
|
|
|||
|
|
@ -1,14 +1,94 @@
|
|||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { describe, expect, it, vi, type MockInstance } from 'vitest'
|
||||
import {
|
||||
cliGateOptions,
|
||||
defaultConcurrency,
|
||||
formatGateResultReason,
|
||||
gatesForMode,
|
||||
parsePidPpidLines,
|
||||
runGate,
|
||||
runGates,
|
||||
taskkillArgs,
|
||||
type Gate,
|
||||
type GateResult,
|
||||
} from './run-gates.ts'
|
||||
|
||||
/**
|
||||
* Capture output a gate streams through runGate's streamOutput path.
|
||||
* @returns the accumulated chunks and the stdout spy to restore in finally.
|
||||
*/
|
||||
function captureStreamedOutput(): { writes: string[]; write: MockInstance } {
|
||||
const writes: string[] = []
|
||||
const write = vi.spyOn(process.stdout, 'write').mockImplementation((chunk) => {
|
||||
writes.push(String(chunk))
|
||||
return true
|
||||
})
|
||||
return { writes, write }
|
||||
}
|
||||
|
||||
/**
|
||||
* A process has stopped executing when its /proc entry is gone, or when it
|
||||
* lingers as a zombie ('Z') — an un-reaped but dead entry still answers
|
||||
* kill(pid, 0), so existence is not a liveness check. Non-Linux falls back to
|
||||
* kill(pid, 0), whose ESRCH means the process is gone.
|
||||
*/
|
||||
function procStopped(pid: number): boolean {
|
||||
if (process.platform === 'linux') {
|
||||
try {
|
||||
const stat = readFileSync(`/proc/${pid}/stat`, 'utf8')
|
||||
return /\)\s+Z\s/.test(stat)
|
||||
} catch {
|
||||
return true
|
||||
}
|
||||
}
|
||||
try {
|
||||
process.kill(pid, 0)
|
||||
return false
|
||||
} catch {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until the captured output contains `marker` and the grandchild pid the
|
||||
* gate printed, then return that pid.
|
||||
* @param writes - chunks captured from the gate's streamed stdout.
|
||||
* @param marker - the output line that proves the gate reached the abort point.
|
||||
* @param deadline - fail the wait when exceeded.
|
||||
* @returns the grandchild pid printed by the gate script.
|
||||
*/
|
||||
async function waitForGrandchildPid(writes: string[], marker: string, deadline: number): Promise<number> {
|
||||
let pid: number | undefined
|
||||
while ((pid === undefined || !writes.join('').includes(marker)) && Date.now() < deadline) {
|
||||
const match = writes.join('').match(/grandchild:(\d+)/)
|
||||
if (match !== null) pid = Number(match[1])
|
||||
await new Promise(resolve => setTimeout(resolve, 50))
|
||||
}
|
||||
expect(pid ?? 0).toBeGreaterThan(0)
|
||||
expect(writes.join('')).toContain(marker)
|
||||
return pid!
|
||||
}
|
||||
|
||||
/**
|
||||
* Abort the run and assert it settles marked aborted with the grandchild no
|
||||
* longer executing — the abort path must have signalled it from the captured
|
||||
* descendant list rather than settling over a live orphan.
|
||||
* @param promise - the pending `runGate` promise.
|
||||
* @param controller - the signal source to abort.
|
||||
* @param pid - the grandchild pid the gate script printed.
|
||||
*/
|
||||
async function abortAndExpectTreeStopped(promise: Promise<GateResult>, controller: AbortController, pid: number): Promise<void> {
|
||||
controller.abort()
|
||||
const result = await promise
|
||||
expect(result.aborted).toBe(true)
|
||||
const stopDeadline = Date.now() + 8000
|
||||
while (!procStopped(pid) && Date.now() < stopDeadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 50))
|
||||
}
|
||||
expect(procStopped(pid)).toBe(true)
|
||||
}
|
||||
|
||||
|
||||
function gate(id: string, options: Partial<Gate> = {}): Gate {
|
||||
return {
|
||||
id,
|
||||
|
|
@ -294,7 +374,7 @@ describe('gate graph validation', () => {
|
|||
const results = await runGates([dependent, root], 1, execute)
|
||||
|
||||
expect(execute).toHaveBeenCalledOnce()
|
||||
expect(execute).toHaveBeenCalledWith(root)
|
||||
expect(execute).toHaveBeenCalledWith(root, undefined)
|
||||
expect(results[0]).toMatchObject({ gate: dependent, status: 'skipped', error: 'dependency failed or skipped: root' })
|
||||
})
|
||||
|
||||
|
|
@ -529,3 +609,300 @@ describe('gate process outcomes', () => {
|
|||
expect(formatGateResultReason(result)).toBe('signal SIGTERM')
|
||||
})
|
||||
})
|
||||
|
||||
describe('fail-fast scheduling', () => {
|
||||
it('aborts the aggregate at the first blocking failure', async () => {
|
||||
const slow = gate('slow')
|
||||
const fast = gate('fast')
|
||||
const dependent = gate('dependent', { needs: ['slow'] })
|
||||
const execute = vi.fn(async (subject: Gate, signal?: AbortSignal) => {
|
||||
if (subject.id === 'fast') {
|
||||
return new Promise<GateResult>((resolve) => {
|
||||
signal?.addEventListener('abort', () => {
|
||||
// The real runGate marks a gate the abort terminated; the drain
|
||||
// must then record it skipped rather than keep the failure.
|
||||
resolve({ ...resultFor(subject, 'failed'), aborted: true })
|
||||
}, { once: true })
|
||||
})
|
||||
}
|
||||
return resultFor(subject, subject.id === 'slow' ? 'failed' : 'passed')
|
||||
})
|
||||
|
||||
const results = await runGates([slow, fast, dependent], 2, execute, () => {}, { failFast: true })
|
||||
|
||||
expect(execute.mock.calls.map(([subject]) => subject.id)).toEqual(['slow', 'fast'])
|
||||
expect(results.map(result => result.status)).toEqual(['failed', 'skipped', 'skipped'])
|
||||
expect(results[1]).toMatchObject({
|
||||
status: 'skipped',
|
||||
error: 'aborted by fail-fast: slow failed',
|
||||
})
|
||||
expect(results[2]).toMatchObject({
|
||||
status: 'skipped',
|
||||
error: 'aborted by fail-fast: slow failed',
|
||||
})
|
||||
})
|
||||
|
||||
it('does not abort on a non-blocking gate failure', async () => {
|
||||
const observational = gate('observational', { allowFailure: true })
|
||||
const root = gate('root')
|
||||
const execute = vi.fn(async (subject: Gate) => (
|
||||
resultFor(subject, subject.id === 'observational' ? 'failed' : 'passed')
|
||||
))
|
||||
|
||||
const results = await runGates([observational, root], 2, execute, () => {}, { failFast: true })
|
||||
|
||||
expect(execute).toHaveBeenCalledTimes(2)
|
||||
expect(results.map(result => result.status)).toEqual(['failed', 'passed'])
|
||||
})
|
||||
|
||||
it('runs independent gates to completion when fail-fast is disabled', async () => {
|
||||
const root = gate('root')
|
||||
const sibling = gate('sibling')
|
||||
const execute = vi.fn(async (subject: Gate) => (
|
||||
resultFor(subject, subject.id === 'root' ? 'failed' : 'passed')
|
||||
))
|
||||
|
||||
const results = await runGates([root, sibling], 2, execute, () => {}, { failFast: false })
|
||||
|
||||
expect(execute).toHaveBeenCalledTimes(2)
|
||||
expect(results.map(result => result.status)).toEqual(['failed', 'passed'])
|
||||
})
|
||||
|
||||
it('kills the child when the abort signal fires', async () => {
|
||||
const controller = new AbortController()
|
||||
const promise = runGate(gate('killable', { args: ['-e', 'setInterval(() => {}, 1000)'] }), controller.signal)
|
||||
controller.abort()
|
||||
const result = await promise
|
||||
|
||||
expect(result.status).toBe('failed')
|
||||
expect(result.aborted).toBe(true)
|
||||
if (process.platform !== 'win32') expect(result.signalCode).toBe('SIGTERM')
|
||||
})
|
||||
|
||||
it.skipIf(process.platform === 'win32')('marks a zero-exit child as aborted when the signal fired', async () => {
|
||||
const { writes, write } = captureStreamedOutput()
|
||||
try {
|
||||
const controller = new AbortController()
|
||||
const child = gate('traps-signal', {
|
||||
args: ['-e', "process.stdout.write('ready\\n'); process.on('SIGTERM', () => process.exit(0)); setInterval(() => {}, 1000)"],
|
||||
streamOutput: true,
|
||||
})
|
||||
const promise = runGate(child, controller.signal)
|
||||
// Wait for the child to register its SIGTERM trap before aborting, so
|
||||
// the signal is caught and the child really exits zero.
|
||||
const deadline = Date.now() + 5000
|
||||
while (!writes.join('').includes('ready') && Date.now() < deadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 10))
|
||||
}
|
||||
controller.abort()
|
||||
const result = await promise
|
||||
|
||||
// The child trapped the signal and exited zero; the drain must not
|
||||
// report this gate passed, so the raw outcome carries the abort mark.
|
||||
expect(result.status).toBe('passed')
|
||||
expect(result.aborted).toBe(true)
|
||||
} finally {
|
||||
write.mockRestore()
|
||||
}
|
||||
})
|
||||
|
||||
it.skipIf(process.platform === 'win32')('kills the whole gate process tree when the abort signal fires', async () => {
|
||||
const { writes, write } = captureStreamedOutput()
|
||||
const controller = new AbortController()
|
||||
let promise: Promise<GateResult> | undefined
|
||||
try {
|
||||
const script = [
|
||||
"const { spawn } = require('node:child_process')",
|
||||
// Detached, so the grandchild leads its own process group: the gate
|
||||
// group signal cannot reach it, and only the descendant enumeration in
|
||||
// treeKill does — the shape of a nested run-gates' leaf gates.
|
||||
"const grandchild = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: true })",
|
||||
"process.stdout.write('grandchild:' + grandchild.pid + '\\n')",
|
||||
'setInterval(() => {}, 1000)',
|
||||
].join(';')
|
||||
promise = runGate(gate('tree', { args: ['-e', script], streamOutput: true }), controller.signal)
|
||||
const deadline = Date.now() + 5000
|
||||
let pid: number | undefined
|
||||
while (pid === undefined && Date.now() < deadline) {
|
||||
const match = writes.join('').match(/grandchild:(\d+)/)
|
||||
if (match !== null) pid = Number(match[1])
|
||||
else await new Promise(resolve => setTimeout(resolve, 20))
|
||||
}
|
||||
expect(pid ?? 0).toBeGreaterThan(0)
|
||||
controller.abort()
|
||||
const result = await promise
|
||||
expect(result.status).toBe('failed')
|
||||
// The descendant enumeration signals the detached grandchild at the same
|
||||
// time as the group signal reaches the direct child; the direct child's
|
||||
// own death closes the gate pipes, so poll for the grandchild to stop
|
||||
// executing rather than asserting on a fixed instant.
|
||||
const stopDeadline = Date.now() + 5000
|
||||
while (!procStopped(pid!) && Date.now() < stopDeadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 20))
|
||||
}
|
||||
expect(procStopped(pid!)).toBe(true)
|
||||
} finally {
|
||||
// A failed wait or assertion must not leave the forever-looping detached
|
||||
// grandchild behind on the host: abort the gate and wait for the
|
||||
// process tree to settle before restoring the spy.
|
||||
controller.abort()
|
||||
await promise
|
||||
write.mockRestore()
|
||||
}
|
||||
})
|
||||
|
||||
it('forwards host interruption signals to the abort path', async () => {
|
||||
const slow = gate('slow')
|
||||
const sibling = gate('sibling')
|
||||
const execute = vi.fn(async (subject: Gate, signal?: AbortSignal) => {
|
||||
if (subject.id === 'slow') {
|
||||
return new Promise<GateResult>((resolve) => {
|
||||
signal?.addEventListener('abort', () => {
|
||||
// A child can trap the signal and exit zero; the drain must still
|
||||
// record the gate skipped so the interrupted run fails.
|
||||
resolve({ ...resultFor(subject, 'passed'), aborted: true })
|
||||
}, { once: true })
|
||||
})
|
||||
}
|
||||
return resultFor(subject)
|
||||
})
|
||||
|
||||
const promise = runGates([slow, sibling], 1, execute, () => {}, { failFast: true, forwardProcessSignals: true })
|
||||
// The first loop iteration starts `slow` synchronously, so its abort
|
||||
// listener is registered before the signal is emitted.
|
||||
process.emit('SIGTERM')
|
||||
const results = await promise
|
||||
|
||||
expect(execute).toHaveBeenCalledOnce()
|
||||
expect(results.map(result => result.status)).toEqual(['skipped', 'skipped'])
|
||||
expect(results[0]).toMatchObject({
|
||||
status: 'skipped',
|
||||
error: 'aborted by fail-fast: host interruption',
|
||||
})
|
||||
})
|
||||
|
||||
it('pairs host signal forwarding with fail-fast at the CLI entrypoint', () => {
|
||||
expect(cliGateOptions(true)).toEqual({ failFast: true, forwardProcessSignals: true })
|
||||
expect(cliGateOptions(false)).toEqual({ failFast: false, forwardProcessSignals: false })
|
||||
})
|
||||
|
||||
it('rejects host signal forwarding without fail-fast', async () => {
|
||||
const execute = vi.fn(async (subject: Gate) => resultFor(subject))
|
||||
|
||||
await expect(runGates([gate('subject')], 1, execute, () => {}, { forwardProcessSignals: true }))
|
||||
.rejects.toThrow('forwardProcessSignals requires failFast')
|
||||
expect(execute).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('leaves an un-aborted child running to completion', async () => {
|
||||
const result = await runGate(gate('settles', { args: ['-e', ''] }), new AbortController().signal)
|
||||
|
||||
expect(result.status).toBe('passed')
|
||||
expect(result.aborted).toBe(false)
|
||||
})
|
||||
|
||||
it.skipIf(process.platform === 'win32')('kills a detached descendant that outlived the child when the abort arrives later', async () => {
|
||||
const writes: string[] = []
|
||||
const write = vi.spyOn(process.stdout, 'write').mockImplementation((chunk) => {
|
||||
writes.push(String(chunk))
|
||||
return true
|
||||
})
|
||||
const controller = new AbortController()
|
||||
let promise: Promise<GateResult> | undefined
|
||||
try {
|
||||
const script = [
|
||||
"const { spawn } = require('node:child_process')",
|
||||
// Detached with inherited stdio: the grandchild leads its own process
|
||||
// group (the gate group signal misses it) and holds the gate's
|
||||
// stdout write end (so `close` stays pending past the child exit).
|
||||
"const grandchild = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: true, stdio: 'inherit' })",
|
||||
"process.stdout.write('grandchild:' + grandchild.pid + '\\n')",
|
||||
// Outlive the first descendant-sampler tick with margin so the cache
|
||||
// holds the grandchild even on a loaded runner, then exit normally
|
||||
// before the abort arrives.
|
||||
"setTimeout(() => { process.stdout.write('child-exit\\n'); process.exit(0) }, 8000)",
|
||||
].join(';')
|
||||
promise = runGate(gate('late-abort', { args: ['-e', script], streamOutput: true }), controller.signal)
|
||||
const pid = await waitForGrandchildPid(writes, 'child-exit', Date.now() + 10000)
|
||||
// terminate must not re-enumerate over the sampler cache now that the
|
||||
// child is gone; the detached grandchild is killed from the cached list.
|
||||
await abortAndExpectTreeStopped(promise, controller, pid)
|
||||
} finally {
|
||||
// A failed wait or assertion must not leave the forever-looping detached
|
||||
// grandchild behind on the host: abort the gate and wait for the
|
||||
// process tree to settle before restoring the spy.
|
||||
controller.abort()
|
||||
await promise
|
||||
write.mockRestore()
|
||||
}
|
||||
}, 20000)
|
||||
|
||||
it.skipIf(process.platform === 'win32')('keeps a reparented detached descendant tracked across a sampler tick', async () => {
|
||||
const { writes, write } = captureStreamedOutput()
|
||||
const controller = new AbortController()
|
||||
let promise: Promise<GateResult> | undefined
|
||||
try {
|
||||
const script = [
|
||||
"const { spawn } = require('node:child_process')",
|
||||
// Wrapper spawns a detached grandchild with inherited stdio (its own
|
||||
// process group, holding the gate's stdout write end), prints the pid,
|
||||
// then exits after 7 seconds — after the first sampler tick, before
|
||||
// the second. From then on the grandchild is reparented and
|
||||
// unreachable by parent id.
|
||||
"const wrapper = spawn(process.execPath, ['-e', \"const { spawn } = require('node:child_process'); const grandchild = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: true, stdio: 'inherit' }); process.stdout.write('grandchild:' + grandchild.pid + '\\\\n'); setTimeout(() => process.exit(0), 7000)\"], { stdio: 'inherit' })",
|
||||
"wrapper.on('exit', () => process.stdout.write('wrapper-exited\\n'))",
|
||||
// Keep the root child alive past the abort with a heartbeat so the
|
||||
// test can abort while it is still running.
|
||||
"setInterval(() => process.stdout.write('hb\\n'), 1000)",
|
||||
].join(';')
|
||||
promise = runGate(gate('sampler-merge', { args: ['-e', script], streamOutput: true }), controller.signal)
|
||||
const pid = await waitForGrandchildPid(writes, 'wrapper-exited', Date.now() + 15000)
|
||||
// Wait past the second sampler tick (t=10) with margin: a replacing tick
|
||||
// would drop the reparented grandchild from the cache, after which the
|
||||
// abort cannot reach it. The root child keeps running throughout.
|
||||
const tickDeadline = Date.now() + 10000
|
||||
const wrapperExitedAt = Date.now()
|
||||
while (Date.now() - wrapperExitedAt < 5000 && Date.now() < tickDeadline) {
|
||||
await new Promise(resolve => setTimeout(resolve, 50))
|
||||
}
|
||||
expect(Date.now() - wrapperExitedAt).toBeGreaterThanOrEqual(5000)
|
||||
await abortAndExpectTreeStopped(promise, controller, pid)
|
||||
} finally {
|
||||
// A failed wait or assertion must not leave the forever-looping detached
|
||||
// grandchild behind on the host: abort the gate and wait for the
|
||||
// process tree to settle before restoring the spy.
|
||||
controller.abort()
|
||||
await promise
|
||||
write.mockRestore()
|
||||
}
|
||||
}, 30000)
|
||||
})
|
||||
|
||||
describe('process-table parsing', () => {
|
||||
it('parses `pid ppid` rows from a POSIX ps dump', () => {
|
||||
expect(parsePidPpidLines(' 123 1\n456 123\n 789 456\n')).toEqual([[123, 1], [456, 123], [789, 456]])
|
||||
})
|
||||
|
||||
it('parses Windows PowerShell Get-CimInstance output of the same shape', () => {
|
||||
expect(parsePidPpidLines(' 123 1\r\n456 123\r\n')).toEqual([[123, 1], [456, 123]])
|
||||
})
|
||||
|
||||
it('drops blank and malformed lines', () => {
|
||||
expect(parsePidPpidLines(' 123 1\n\ncommand not found\n999 abc\n')).toEqual([[123, 1]])
|
||||
})
|
||||
})
|
||||
|
||||
describe('Windows tree termination', () => {
|
||||
it('targets the root first and each captured descendant after it', () => {
|
||||
expect(taskkillArgs(100, [201, 302, 403])).toEqual([
|
||||
['/PID', '100', '/T', '/F'],
|
||||
['/PID', '201', '/T', '/F'],
|
||||
['/PID', '302', '/T', '/F'],
|
||||
['/PID', '403', '/T', '/F'],
|
||||
])
|
||||
})
|
||||
|
||||
it('terminates the root alone when no descendant was captured', () => {
|
||||
expect(taskkillArgs(100, [])).toEqual([['/PID', '100', '/T', '/F']])
|
||||
})
|
||||
})
|
||||
|
|
|
|||
|
|
@ -5,7 +5,8 @@
|
|||
* dependency graphs, scheduler environment, and process diagnostics.
|
||||
* @see ../.agents/notes/implemented/process/2026-07-06-parallel-pre-push-gates.md
|
||||
*/
|
||||
import { spawn } from 'node:child_process'
|
||||
import { spawn, spawnSync } from 'node:child_process'
|
||||
import { readdirSync, readFileSync } from 'node:fs'
|
||||
import { availableParallelism } from 'node:os'
|
||||
import { resolve } from 'node:path'
|
||||
import { performance } from 'node:perf_hooks'
|
||||
|
|
@ -69,6 +70,10 @@ export interface GateResult {
|
|||
exitCode: number | null
|
||||
signalCode: NodeJS.Signals | null
|
||||
error?: string
|
||||
/** True when the shared abort signal terminated this gate before its outcome
|
||||
* was observed; such a result must not be reported as passed, even if the
|
||||
* child trapped the signal and exited zero. */
|
||||
aborted?: boolean
|
||||
}
|
||||
|
||||
interface GateOutputChunk {
|
||||
|
|
@ -86,7 +91,7 @@ interface ConcurrencyDefault {
|
|||
source: string
|
||||
}
|
||||
|
||||
type GateExecutor = (gate: Gate) => Promise<GateResult>
|
||||
type GateExecutor = (gate: Gate, signal?: AbortSignal) => Promise<GateResult>
|
||||
type ResultObserver = (result: GateResult) => void
|
||||
|
||||
const root = resolve(import.meta.dirname, '..')
|
||||
|
|
@ -103,16 +108,28 @@ async function main(args: string[]): Promise<number> {
|
|||
const concurrencySource = concurrencyOverride === undefined || concurrencyOverride === ''
|
||||
? concurrencyDefault.source
|
||||
: '$DSH_GATE_CONCURRENCY'
|
||||
const failFast = flagEnabled('DSH_GATE_FAIL_FAST')
|
||||
const startedAt = performance.now()
|
||||
console.log(`run-gates: ${mode} running ${gates.length} gate(s) with ${maxConcurrency} worker(s) from ${concurrencySource}.`)
|
||||
console.log(`run-gates: ${mode} running ${gates.length} gate(s) with ${maxConcurrency} worker(s) from ${concurrencySource}${failFast ? ', fail-fast after first blocking failure' : ''}.`)
|
||||
|
||||
const results = await runGates(gates, maxConcurrency, runGate, printResult)
|
||||
const results = await runGates(gates, maxConcurrency, runGate, printResult, cliGateOptions(failFast))
|
||||
printSummary(results, performance.now() - startedAt)
|
||||
return results.some(result => result.gate.allowFailure !== true && (result.status === 'failed' || result.status === 'skipped'))
|
||||
? 1
|
||||
: 0
|
||||
}
|
||||
|
||||
/**
|
||||
* The options the CLI entrypoint hands to the scheduler. Host signal
|
||||
* forwarding always follows fail-fast: children are detached only then, so
|
||||
* without it the forwarding would have no tree to drain.
|
||||
* @param failFast - whether `DSH_GATE_FAIL_FAST` is enabled.
|
||||
* @returns the scheduler options for the entrypoint.
|
||||
*/
|
||||
export function cliGateOptions(failFast: boolean): RunGatesOptions {
|
||||
return { failFast, forwardProcessSignals: failFast }
|
||||
}
|
||||
|
||||
function parseMode(raw: string | undefined): Mode {
|
||||
switch (raw) {
|
||||
case 'ci-primary':
|
||||
|
|
@ -836,12 +853,30 @@ function findDependencyCycle(gates: readonly Gate[]): string[] | undefined {
|
|||
return undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Scheduling options for one aggregate.
|
||||
*/
|
||||
export interface RunGatesOptions {
|
||||
/** Stop the aggregate at the first blocking gate failure. */
|
||||
failFast?: boolean
|
||||
/** Forward host SIGINT/SIGTERM to the abort path so detached gate trees are
|
||||
* terminated when the run itself is interrupted or the runner cancels it.
|
||||
* Tree termination additionally requires failFast, because only then is the
|
||||
* abort signal passed to the executor and children detached. */
|
||||
forwardProcessSignals?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate and run one aggregate before the injected executor can start a child.
|
||||
* @param gates - complete aggregate to execute.
|
||||
* @param maxActive - maximum concurrent child count.
|
||||
* @param execute - child-process executor.
|
||||
* @param execute - child-process executor; receives the abort signal only when
|
||||
* fail-fast is enabled, so ordinary runs keep their children in the host
|
||||
* process group.
|
||||
* @param observe - result observer invoked when each gate settles.
|
||||
* @param options - scheduling options; fail-fast aborts the aggregate at the
|
||||
* first blocking gate failure by killing running children and skipping every
|
||||
* not-yet-run gate.
|
||||
* @returns results in aggregate order.
|
||||
*/
|
||||
export async function runGates(
|
||||
|
|
@ -849,54 +884,105 @@ export async function runGates(
|
|||
maxActive: number,
|
||||
execute: GateExecutor,
|
||||
observe: ResultObserver = () => {},
|
||||
options: RunGatesOptions = {},
|
||||
): Promise<GateResult[]> {
|
||||
validateGateGraph(gates)
|
||||
if (!Number.isSafeInteger(maxActive) || maxActive < 1) {
|
||||
throw new Error(`run-gates: max concurrency must be a positive integer, got ${JSON.stringify(maxActive)}.`)
|
||||
}
|
||||
if (options.forwardProcessSignals === true && options.failFast !== true) {
|
||||
throw new Error('run-gates: forwardProcessSignals requires failFast, otherwise no child is detached or killed.')
|
||||
}
|
||||
const states = new Map<string, GateState>(gates.map(gate => [gate.id, 'pending']))
|
||||
const results = new Map<string, GateResult>()
|
||||
const running: RunningGate[] = []
|
||||
|
||||
for (;;) {
|
||||
let madeProgress = false
|
||||
while (running.length < maxActive) {
|
||||
const ready = gates.find(gate => states.get(gate.id) === 'pending' && predecessorsReady(gate, states))
|
||||
if (ready === undefined) break
|
||||
states.set(ready.id, 'running')
|
||||
running.push({ gate: ready, promise: execute(ready) })
|
||||
console.log(`run-gates: start ${ready.label}`)
|
||||
madeProgress = true
|
||||
const abort = new AbortController()
|
||||
let abortCause: string | undefined
|
||||
// Host interruption (terminal Ctrl+C, runner cancellation) drains through
|
||||
// the same abort path as a gate failure, so detached trees are killed and
|
||||
// never orphaned. Handlers are removed before returning.
|
||||
const hostSignals = options.forwardProcessSignals === true ? ['SIGINT', 'SIGTERM'] as const : []
|
||||
const hostHandlers = hostSignals.map((name) => {
|
||||
const handler = () => {
|
||||
abortCause = abortCause ?? 'host interruption'
|
||||
abort.abort()
|
||||
}
|
||||
process.on(name, handler)
|
||||
return { name, handler }
|
||||
})
|
||||
const failFastSignal = options.failFast === true ? abort.signal : undefined
|
||||
|
||||
if (running.length === 0) {
|
||||
const pending = gates.filter(gate => states.get(gate.id) === 'pending')
|
||||
if (pending.length === 0) break
|
||||
const gate = pending.find(item => (item.needs ?? []).some(id => gateFailed(states.get(id))))
|
||||
if (gate === undefined) throw new Error('run-gates: validated graph stalled without a failed dependency.')
|
||||
const failedDeps = (gate.needs ?? []).filter(id => gateFailed(states.get(id)))
|
||||
const result: GateResult = {
|
||||
gate,
|
||||
status: 'skipped',
|
||||
durationMs: 0,
|
||||
output: [],
|
||||
exitCode: null,
|
||||
signalCode: null,
|
||||
error: `dependency failed or skipped: ${failedDeps.join(', ')}`,
|
||||
try {
|
||||
for (;;) {
|
||||
let madeProgress = false
|
||||
if (abortCause === undefined) {
|
||||
while (running.length < maxActive) {
|
||||
const ready = gates.find(gate => states.get(gate.id) === 'pending' && predecessorsReady(gate, states))
|
||||
if (ready === undefined) break
|
||||
states.set(ready.id, 'running')
|
||||
running.push({ gate: ready, promise: execute(ready, failFastSignal) })
|
||||
console.log(`run-gates: start ${ready.label}`)
|
||||
madeProgress = true
|
||||
}
|
||||
}
|
||||
states.set(gate.id, 'skipped')
|
||||
results.set(gate.id, result)
|
||||
observe(result)
|
||||
continue
|
||||
}
|
||||
|
||||
if (!madeProgress) {
|
||||
const settled = await Promise.race(running.map(async item => ({ item, result: await item.promise })))
|
||||
running.splice(running.indexOf(settled.item), 1)
|
||||
states.set(settled.item.gate.id, settled.result.status)
|
||||
results.set(settled.item.gate.id, settled.result)
|
||||
observe(settled.result)
|
||||
if (running.length === 0) {
|
||||
if (abortCause !== undefined) {
|
||||
for (const gate of gates) {
|
||||
if (states.get(gate.id) !== 'pending') continue
|
||||
const skipped = skippedByFailFast(gate, abortCause)
|
||||
states.set(gate.id, 'skipped')
|
||||
results.set(gate.id, skipped)
|
||||
observe(skipped)
|
||||
}
|
||||
break
|
||||
}
|
||||
const pending = gates.filter(gate => states.get(gate.id) === 'pending')
|
||||
if (pending.length === 0) break
|
||||
const gate = pending.find(item => (item.needs ?? []).some(id => gateFailed(states.get(id))))
|
||||
if (gate === undefined) throw new Error('run-gates: validated graph stalled without a failed dependency.')
|
||||
const failedDeps = (gate.needs ?? []).filter(id => gateFailed(states.get(id)))
|
||||
const result: GateResult = {
|
||||
gate,
|
||||
status: 'skipped',
|
||||
durationMs: 0,
|
||||
output: [],
|
||||
exitCode: null,
|
||||
signalCode: null,
|
||||
error: `dependency failed or skipped: ${failedDeps.join(', ')}`,
|
||||
}
|
||||
states.set(gate.id, 'skipped')
|
||||
results.set(gate.id, result)
|
||||
observe(result)
|
||||
continue
|
||||
}
|
||||
|
||||
if (!madeProgress) {
|
||||
const settled = await Promise.race(running.map(async item => ({ item, result: await item.promise })))
|
||||
running.splice(running.indexOf(settled.item), 1)
|
||||
const observed = abortCause === undefined || settled.result.aborted !== true
|
||||
? settled.result
|
||||
: skippedByFailFast(settled.item.gate, abortCause)
|
||||
states.set(settled.item.gate.id, observed.status)
|
||||
results.set(settled.item.gate.id, observed)
|
||||
observe(observed)
|
||||
if (abortCause === undefined && options.failFast === true
|
||||
&& observed.status === 'failed' && settled.item.gate.allowFailure !== true) {
|
||||
abortCause = `${observed.gate.label} failed`
|
||||
abort.abort()
|
||||
console.error(`run-gates: fail-fast aborting: ${abortCause}.`)
|
||||
for (const gate of gates) {
|
||||
if (states.get(gate.id) !== 'pending') continue
|
||||
const skipped = skippedByFailFast(gate, abortCause)
|
||||
states.set(gate.id, 'skipped')
|
||||
results.set(gate.id, skipped)
|
||||
observe(skipped)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
for (const { name, handler } of hostHandlers) process.removeListener(name, handler)
|
||||
}
|
||||
|
||||
return gates.map((gate) => {
|
||||
|
|
@ -906,6 +992,31 @@ export async function runGates(
|
|||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* The result of a gate that produced no evidence because fail-fast aborted.
|
||||
* A gate whose process settled before the abort took effect keeps its real
|
||||
* result instead: it did produce evidence, and the summary must say so. Any
|
||||
* result settling after the abort — including a genuine independent failure
|
||||
* in the race window, and a child that trapped the signal and exited zero —
|
||||
* is recorded skipped with its partial output discarded, because on Windows a
|
||||
* killed process is indistinguishable from a failed one by exit code alone.
|
||||
* @param gate - the gate that produced no evidence.
|
||||
* @param cause - the full clause naming what aborted the aggregate, e.g.
|
||||
* `typecheck failed` or `host interruption`.
|
||||
* @returns the skipped record with the fail-fast error.
|
||||
*/
|
||||
function skippedByFailFast(gate: Gate, cause: string): GateResult {
|
||||
return {
|
||||
gate,
|
||||
status: 'skipped',
|
||||
durationMs: 0,
|
||||
output: [],
|
||||
exitCode: null,
|
||||
signalCode: null,
|
||||
error: `aborted by fail-fast: ${cause}`,
|
||||
}
|
||||
}
|
||||
|
||||
function predecessorsReady(gate: Gate, states: Map<string, GateState>): boolean {
|
||||
return (gate.needs ?? []).every(id => states.get(id) === 'passed')
|
||||
&& (gate.after ?? []).every(id => gateSettled(states.get(id)))
|
||||
|
|
@ -922,12 +1033,17 @@ function gateFailed(state: GateState | undefined): boolean {
|
|||
/**
|
||||
* Execute one gate through the real shell-free child-process boundary.
|
||||
* @param gate - command and scheduler environment to execute.
|
||||
* @param signal - abort signal that terminates the whole gate process tree when
|
||||
* the aggregate fails fast; an already-aborted signal terminates it
|
||||
* immediately. A provided signal spawns the child detached so POSIX can signal
|
||||
* its process group and Windows can reach its tree through taskkill.
|
||||
* @returns the complete process outcome.
|
||||
*/
|
||||
export async function runGate(gate: Gate): Promise<GateResult> {
|
||||
export async function runGate(gate: Gate, signal?: AbortSignal): Promise<GateResult> {
|
||||
const started = performance.now()
|
||||
const output: GateOutputChunk[] = []
|
||||
let spawnError: string | undefined
|
||||
let aborted = false
|
||||
|
||||
const outcome = await new Promise<{
|
||||
exitCode: number | null
|
||||
|
|
@ -937,6 +1053,7 @@ export async function runGate(gate: Gate): Promise<GateResult> {
|
|||
cwd: root,
|
||||
env: { ...process.env, ...gate.env },
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
detached: signal !== undefined && process.platform !== 'win32',
|
||||
})
|
||||
child.stdout.setEncoding('utf8')
|
||||
child.stderr.setEncoding('utf8')
|
||||
|
|
@ -948,11 +1065,187 @@ export async function runGate(gate: Gate): Promise<GateResult> {
|
|||
if (gate.streamOutput === true) process.stderr.write(chunk)
|
||||
else output.push({ stream: 'stderr', text: chunk })
|
||||
})
|
||||
// Deliver one signal to the entire gate tree: the negative pid targets the
|
||||
// POSIX process group the detached child leads; Windows has no groups, so
|
||||
// taskkill walks the tree rooted at the child and force-terminates (a
|
||||
// taskkill without `/F` does not terminate console processes, which is
|
||||
// what gate commands are). Outcomes are deliberately unchecked because
|
||||
// delivery races tree exit, and a missing taskkill binary is as tolerable
|
||||
// as ESRCH. Mirrors the subprocess package's teardown contract
|
||||
// (packages/subprocess/subprocess-local/src/spawn.ts).
|
||||
const treeKill = (signalToSend: 'SIGTERM' | 'SIGKILL') => {
|
||||
const pid = child.pid
|
||||
if (pid === undefined) return
|
||||
if (process.platform === 'win32') {
|
||||
for (const args of taskkillArgs(pid, descendants)) {
|
||||
spawnSync('taskkill', args, { stdio: 'ignore' })
|
||||
}
|
||||
return
|
||||
}
|
||||
try {
|
||||
process.kill(-pid, signalToSend)
|
||||
} catch {
|
||||
// The group is gone; the direct child may still be alive alone.
|
||||
child.kill(signalToSend)
|
||||
}
|
||||
// The captured list stays valid after the group kill reparents the
|
||||
// detached descendants of a nested run-gates (the `check:node-compat`
|
||||
// and `check:ci:lint:contracts-ready` gates in ci-consumers): pids do
|
||||
// not change on reparenting, so the escalation reaches leaves that
|
||||
// ignored SIGTERM without re-enumerating.
|
||||
for (const descendantPid of descendants) {
|
||||
try {
|
||||
process.kill(descendantPid, signalToSend)
|
||||
} catch {
|
||||
// The descendant exited between the enumeration and the signal.
|
||||
}
|
||||
}
|
||||
}
|
||||
let escalation: ReturnType<typeof setTimeout> | undefined
|
||||
let terminatedAt = 0
|
||||
// Captured once at terminate and re-signalled on escalation: the group
|
||||
// kill reaps the direct child, after which its detached descendants are
|
||||
// reparented and unreachable by parent id, so the escalation cannot
|
||||
// re-enumerate them.
|
||||
let descendants: number[] = []
|
||||
let pipeDrain: ReturnType<typeof setTimeout> | undefined
|
||||
const terminate = () => {
|
||||
aborted = true
|
||||
const pid = child.pid
|
||||
// Merge while the child is still alive: re-enumerating alone would drop
|
||||
// a descendant that an exited intermediate reparented out of the parent
|
||||
// chain, and replacing the list entirely would lose the sampler's
|
||||
// last-known entries when the child already exited. Union preserves both.
|
||||
// The sampler runs on every platform (including Windows, where an
|
||||
// exited intermediate's table record vanishes and a fresh enumeration
|
||||
// cannot cross the gap), so the cache is the source of truth once the
|
||||
// child is gone.
|
||||
if (pid !== undefined && child.exitCode === null && child.signalCode === null) {
|
||||
descendants = [...new Set([...descendants, ...descendantPids(pid)])]
|
||||
}
|
||||
treeKill('SIGTERM')
|
||||
if (escalation === undefined) {
|
||||
terminatedAt = Date.now()
|
||||
// Force-kill at the deadline regardless of the direct child's exit
|
||||
// state: when the wrapper dies but a grandchild ignores SIGTERM and
|
||||
// still holds the stdio pipes, `close` has not fired and the tree must
|
||||
// still be killed. treeKill swallows an already-absent group.
|
||||
escalation = setTimeout(() => { treeKill('SIGKILL') }, 5000)
|
||||
}
|
||||
if (pipeDrain === undefined) {
|
||||
// `close` can stay pending past the direct child's exit when a
|
||||
// descendant holds the stdio write ends (escaped process group, or
|
||||
// uninterruptible I/O that keeps the SIGKILL pending). Bound the wait
|
||||
// past the 5-second SIGKILL grace and force the streams closed so
|
||||
// fail-fast settles instead of hanging to the job timeout. Only the
|
||||
// abort path arms it: on an ordinary run a gate that outlives its
|
||||
// descendants must keep waiting rather than report passed over a live
|
||||
// leak. Armed in terminate (not only at `exit`) so the window where
|
||||
// the child already exited before the abort is covered too.
|
||||
pipeDrain = setTimeout(() => {
|
||||
child.stdout.destroy()
|
||||
child.stderr.destroy()
|
||||
child.stdin.destroy()
|
||||
}, 10000)
|
||||
}
|
||||
}
|
||||
if (signal !== undefined) {
|
||||
if (signal.aborted) terminate()
|
||||
else signal.addEventListener('abort', terminate, { once: true })
|
||||
}
|
||||
// Refresh the descendant cache while the child runs, so an abort that
|
||||
// arrives after the child already exited can still reach a detached
|
||||
// descendant the child left behind: once the child is gone, its
|
||||
// descendants are reparented (POSIX) or their intermediate's table record
|
||||
// is gone (Windows), so a fresh enumeration cannot cross the gap. The
|
||||
// cache is primed at spawn and refreshed every 5 seconds, so a descendant
|
||||
// is captured once it appears in any enumeration whose parent chain is
|
||||
// still fully present in the table; the residual window is a descendant
|
||||
// that never appears in such a snapshot — created after one enumeration
|
||||
// and orphaned before the next. Enumeration is asynchronous (a slow
|
||||
// WMI/CIM call is bounded by its own 10-second timeout), so a gate's
|
||||
// output draining and exit handling are never blocked while the sampler
|
||||
// reads the process table. Fail-fast runs only; ordinary runs never
|
||||
// abort.
|
||||
let descendantSampler: ReturnType<typeof setInterval> | undefined
|
||||
if (signal !== undefined) {
|
||||
let enumerationInFlight: { cancel: () => void } | undefined
|
||||
const refreshDescendants = () => {
|
||||
const pid = child.pid
|
||||
if (pid === undefined || child.exitCode !== null || child.signalCode !== null) return
|
||||
if (enumerationInFlight !== undefined) return
|
||||
const handle = descendantPidsAsync(pid, process.platform)
|
||||
enumerationInFlight = handle
|
||||
void handle.promise.then((fresh) => {
|
||||
if (enumerationInFlight === handle) enumerationInFlight = undefined
|
||||
// Merge regardless of the child's exit state: the enumeration
|
||||
// started while the child was alive, so its snapshot is the last
|
||||
// reliable view of the tree. The child may exit (its intermediate
|
||||
// gone, its table record vanished) before the promise settles while
|
||||
// a grandchild still holds the stdio write ends and keeps `close`
|
||||
// pending — exactly when terminate needs this list.
|
||||
// Merge instead of replacing, like terminate: an intermediate that
|
||||
// exited since the last tick reparented its detached descendants
|
||||
// out of the parent chain, so a fresh enumeration alone would drop
|
||||
// them. Filter the cache to the still-executing so a long gate
|
||||
// does not accumulate stale pids; while sampler ticks still run the
|
||||
// live filter also keeps the escalation from signalling a reused
|
||||
// pid, but once ticks stop (child exited) the cache can go stale,
|
||||
// and a pid reused after that is the accepted sampling window.
|
||||
descendants = [...new Set([...descendants.filter(processAlive), ...fresh])]
|
||||
})
|
||||
}
|
||||
const cancelInFlightEnumeration = () => {
|
||||
if (enumerationInFlight !== undefined) enumerationInFlight.cancel()
|
||||
enumerationInFlight = undefined
|
||||
}
|
||||
refreshDescendants()
|
||||
descendantSampler = setInterval(refreshDescendants, 5000)
|
||||
// A gate that settles while an enumeration is still running must not
|
||||
// leave the PowerShell subprocess holding stdio handles until its own
|
||||
// timeout: stop it as soon as the child's outcome is known.
|
||||
child.once('close', cancelInFlightEnumeration)
|
||||
child.once('error', cancelInFlightEnumeration)
|
||||
}
|
||||
child.on('error', (error) => {
|
||||
if (escalation !== undefined) clearTimeout(escalation)
|
||||
if (pipeDrain !== undefined) clearTimeout(pipeDrain)
|
||||
if (descendantSampler !== undefined) clearInterval(descendantSampler)
|
||||
if (signal !== undefined) signal.removeEventListener('abort', terminate)
|
||||
spawnError = `failed to start command: ${error.message}`
|
||||
resolveExit({ exitCode: null, signalCode: null })
|
||||
})
|
||||
child.on('close', (exitCode, signalCode) => {
|
||||
if (pipeDrain !== undefined) clearTimeout(pipeDrain)
|
||||
if (descendantSampler !== undefined) clearInterval(descendantSampler)
|
||||
if (signal !== undefined) signal.removeEventListener('abort', terminate)
|
||||
if (escalation !== undefined && process.platform !== 'win32') {
|
||||
// `close` only means the direct child's stdio closed; a grandchild
|
||||
// that ignored SIGTERM and redirected its stdio can outlive it. Do
|
||||
// not settle until the process group and the captured descendants are
|
||||
// confirmed gone — the deadline SIGKILL covers members still alive at
|
||||
// the grace end — so runGate returns only once the tree is quiescent.
|
||||
const confirmGroupGone = () => {
|
||||
if (!groupAlive(child.pid) && descendants.every(descendantPid => !processAlive(descendantPid))) {
|
||||
clearTimeout(escalation)
|
||||
resolveExit({ exitCode, signalCode })
|
||||
return
|
||||
}
|
||||
if (Date.now() - terminatedAt < 8000) {
|
||||
setTimeout(confirmGroupGone, 50)
|
||||
return
|
||||
}
|
||||
// The grace ended with members still alive (e.g. uninterruptible
|
||||
// I/O that even SIGKILL cannot cut). Fail loud instead of reporting
|
||||
// a quiescent tree: the gate is recorded failed either way.
|
||||
console.error(`run-gates: gate tree not quiescent after 8s (${gate.label}).`)
|
||||
clearTimeout(escalation)
|
||||
resolveExit({ exitCode, signalCode })
|
||||
}
|
||||
confirmGroupGone()
|
||||
return
|
||||
}
|
||||
if (escalation !== undefined) clearTimeout(escalation)
|
||||
resolveExit({ exitCode, signalCode })
|
||||
})
|
||||
child.stdin.end()
|
||||
|
|
@ -968,10 +1261,255 @@ export async function runGate(gate: Gate): Promise<GateResult> {
|
|||
exitCode,
|
||||
signalCode,
|
||||
}
|
||||
result.aborted = aborted
|
||||
if (spawnError !== undefined) result.error = spawnError
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse the state, parent, and process-group fields from a `/proc/<pid>/stat`
|
||||
* line. The comm field may contain spaces and parentheses, so the state starts
|
||||
* after the last closing parenthesis.
|
||||
* @param stat - one `/proc/<pid>/stat` line.
|
||||
* @returns state, parent pid, and process-group pid; undefined when truncated.
|
||||
*/
|
||||
function procStatFields(stat: string): { state: string; ppid: number; pgrp: number } | undefined {
|
||||
const fields = stat.slice(stat.lastIndexOf(')') + 2).split(' ')
|
||||
const state = fields[0]
|
||||
const ppid = fields[1]
|
||||
const pgrp = fields[2]
|
||||
if (state === undefined || ppid === undefined || pgrp === undefined) return undefined
|
||||
return { state, ppid: Number(ppid), pgrp: Number(pgrp) }
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether one process is still executing. Zombies (state `Z`) do not count:
|
||||
* they are dead records awaiting reaping, and kill(pid, 0) would report them
|
||||
* as alive. Linux reads /proc/<pid>/stat to distinguish; other platforms fall
|
||||
* back to the signal probe.
|
||||
* @param pid - the process to probe.
|
||||
*/
|
||||
function processAlive(pid: number): boolean {
|
||||
if (process.platform === 'linux') {
|
||||
try {
|
||||
const parsed = procStatFields(readFileSync(`/proc/${pid}/stat`, 'utf8'))
|
||||
return parsed !== undefined && parsed.state !== 'Z'
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
try {
|
||||
process.kill(pid, 0)
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether any member of the child's POSIX process group is still executing.
|
||||
* Zombie entries (state `Z`) do not count: they are dead records awaiting
|
||||
* reaping, and the kill(-pid, 0) group probe would report them as alive.
|
||||
* Linux enumerates /proc to distinguish after a fast-path group probe; other
|
||||
* POSIX platforms fall back to the probe alone.
|
||||
* @param pid - the group leader's pid; undefined or non-positive means the
|
||||
* spawn failed and nothing is alive.
|
||||
*/
|
||||
function groupAlive(pid: number | undefined): boolean {
|
||||
if (pid === undefined || pid <= 0) return false
|
||||
if (process.platform === 'linux') {
|
||||
try {
|
||||
process.kill(-pid, 0)
|
||||
} catch {
|
||||
// ESRCH: the group has no entries at all.
|
||||
return false
|
||||
}
|
||||
try {
|
||||
for (const entry of readdirSync('/proc')) {
|
||||
if (!/^\d+$/.test(entry)) continue
|
||||
try {
|
||||
const parsed = procStatFields(readFileSync(`/proc/${entry}/stat`, 'utf8'))
|
||||
if (parsed !== undefined && parsed.pgrp === pid && parsed.state !== 'Z') return true
|
||||
} catch {
|
||||
// The process exited mid-scan; it is not a live member.
|
||||
}
|
||||
}
|
||||
return false
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
try {
|
||||
process.kill(-pid, 0)
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The pids of every transitive descendant of `root`, read from the live
|
||||
* process table. Linux walks /proc/<pid>/stat parent fields; other platforms
|
||||
* parse `ps` (POSIX) or the CIM process table (Windows) output. This is one
|
||||
* snapshot, not the full tree-ownership mechanism: terminate and the sampler
|
||||
* rely on the 5-second cache to cross an intermediate that exited between
|
||||
* ticks (reparented on POSIX, table record gone on Windows), so a single
|
||||
* enumeration reaches only the descendants whose parent chain is still fully
|
||||
* present in the table.
|
||||
* @param root - the pid whose descendants are wanted.
|
||||
* @returns descendant pids in breadth-first order; empty on enumeration failure.
|
||||
*/
|
||||
function descendantPids(root: number): number[] {
|
||||
if (root <= 0) return []
|
||||
if (process.platform === 'linux') {
|
||||
const rows: Array<[number, number]> = []
|
||||
try {
|
||||
for (const entry of readdirSync('/proc')) {
|
||||
if (!/^\d+$/.test(entry)) continue
|
||||
try {
|
||||
const parsed = procStatFields(readFileSync(`/proc/${entry}/stat`, 'utf8'))
|
||||
if (parsed !== undefined) rows.push([Number(entry), parsed.ppid])
|
||||
} catch {
|
||||
// The process exited mid-scan; skip it.
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
return []
|
||||
}
|
||||
return collectDescendants(root, rows)
|
||||
}
|
||||
let ps: { error?: Error; stdout: string }
|
||||
if (process.platform === 'win32') {
|
||||
// taskkill /T covers the tree only while the root is alive; once the
|
||||
// direct child exits (a descendant still holding the stdio write ends
|
||||
// keeps `close` pending), abort must reach the survivors from a fresh
|
||||
// enumeration. Windows keeps the exited parent's pid in its descendants'
|
||||
// parent column, so this walk still finds the whole tree. A hung
|
||||
// PowerShell (WMI/CIM service trouble) must not stall the abort path
|
||||
// indefinitely, so the enumeration is bounded.
|
||||
ps = spawnSync('powershell', processTableArgs('win32'), { encoding: 'utf8', timeout: 10000 })
|
||||
} else {
|
||||
ps = spawnSync('ps', processTableArgs('posix'), { encoding: 'utf8' })
|
||||
}
|
||||
if (ps.error !== undefined) return []
|
||||
return collectDescendants(root, parsePidPpidLines(ps.stdout))
|
||||
}
|
||||
|
||||
/**
|
||||
* The process-table enumeration command for one platform. Windows queries the
|
||||
* CIM provider through PowerShell (each line `pid ppid`); other platforms use
|
||||
* `ps -axo pid=,ppid=`.
|
||||
* @param platform - the target platform.
|
||||
* @returns the command arguments to enumerate every live process's pid/ppid.
|
||||
*/
|
||||
function processTableArgs(platform: 'win32' | 'posix'): string[] {
|
||||
if (platform === 'win32') {
|
||||
return ['-NoProfile', '-NonInteractive', '-Command', 'Get-CimInstance Win32_Process | ForEach-Object { "$($_.ProcessId) $($_.ParentProcessId)" }']
|
||||
}
|
||||
return ['-axo', 'pid=,ppid=']
|
||||
}
|
||||
|
||||
/**
|
||||
* Asynchronous descendant enumeration, so a slow WMI/CIM call (bounded by a
|
||||
* 10-second timeout) cannot block the event loop: the sampler runs it while
|
||||
* the gate's output streams and exit handling must keep flowing. Returns the
|
||||
* same descendant list as {@link descendantPids}; used by the fail-fast
|
||||
* sampler only, never on the abort path (which needs the synchronous walk to
|
||||
* capture the tree before any member exits).
|
||||
* @param root - the pid whose descendants are wanted.
|
||||
* @param platform - the platform whose table the enumeration reads.
|
||||
* @returns a promise of descendant pids in breadth-first order; empty on
|
||||
* enumeration failure.
|
||||
*/
|
||||
function descendantPidsAsync(root: number, platform: NodeJS.Platform): { promise: Promise<number[]>; cancel: () => void } {
|
||||
if (root <= 0 || platform === 'linux') {
|
||||
// The /proc walk is synchronous inside the async wrapper so the sampler
|
||||
// keeps the same contract on every platform; /proc reads are fast and
|
||||
// need no subprocess, and a completed enumeration needs no cancellation.
|
||||
return { promise: Promise.resolve(descendantPids(root)), cancel: () => {} }
|
||||
}
|
||||
const [command, args] = platform === 'win32'
|
||||
? ['powershell', processTableArgs('win32')]
|
||||
: ['ps', processTableArgs('posix')]
|
||||
const child = spawn(command, args, {
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
timeout: platform === 'win32' ? 10000 : undefined,
|
||||
})
|
||||
child.stdout.setEncoding('utf8')
|
||||
let stdout = ''
|
||||
let settled = false
|
||||
let settle!: (value: number[]) => void
|
||||
const promise = new Promise<number[]>((resolve) => { settle = resolve })
|
||||
const finish = (value: number[]) => {
|
||||
if (settled) return
|
||||
settled = true
|
||||
// The enumeration completed (or was cancelled): stop the subprocess so
|
||||
// the gate does not wait on its stdio handles.
|
||||
child.kill('SIGTERM')
|
||||
settle(value)
|
||||
}
|
||||
child.stdout.on('data', (chunk: string) => { stdout += chunk })
|
||||
child.on('error', () => { finish([]) })
|
||||
child.on('close', () => { finish(collectDescendants(root, parsePidPpidLines(stdout))) })
|
||||
return {
|
||||
promise,
|
||||
cancel: () => { finish([]) },
|
||||
}
|
||||
}
|
||||
|
||||
/** Parse `pid ppid` rows from a process-table dump. Both the POSIX `ps -axo
|
||||
* pid=,ppid=` output and the Windows PowerShell `Get-CimInstance Win32_Process`
|
||||
* projection emit one `pid ppid` pair per line.
|
||||
* @param output - the raw dump text.
|
||||
* @returns the parsed pid/ppid rows in line order; blank and malformed lines
|
||||
* are dropped.
|
||||
*/
|
||||
export function parsePidPpidLines(output: string): Array<[number, number]> {
|
||||
const rows: Array<[number, number]> = []
|
||||
for (const line of output.split('\n')) {
|
||||
const match = line.trim().match(/^(\d+)\s+(\d+)$/)
|
||||
if (match !== null) rows.push([Number(match[1]), Number(match[2])])
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
/**
|
||||
* The taskkill invocations that terminate one Windows gate tree. The direct
|
||||
* child leads, because a live `taskkill /T` walks its whole subtree in one
|
||||
* call; each captured descendant follows individually, because when the root
|
||||
* already exited (a descendant holding the stdio write ends keeps `close`
|
||||
* pending) `taskkill /T` rooted at the dead pid finds nothing — Windows never
|
||||
* reparents, so the ppid chain captured at terminate still reaches the whole
|
||||
* tree, and `/T` lets a surviving intermediate carry its own subtree. A pid
|
||||
* that exited between capture and termination is as tolerable as ESRCH on
|
||||
* POSIX: taskkill reports a nonzero status that is deliberately unchecked.
|
||||
* @param rootPid - the direct child's pid.
|
||||
* @param descendants - the captured descendant pids.
|
||||
* @returns one `taskkill` argument list per pid, in termination order.
|
||||
*/
|
||||
export function taskkillArgs(rootPid: number, descendants: number[]): string[][] {
|
||||
return [rootPid, ...descendants].map(pid => ['/PID', String(pid), '/T', '/F'])
|
||||
}
|
||||
|
||||
/** Breadth-first walk of the pid/ppid rows starting at `root`. */
|
||||
function collectDescendants(root: number, rows: Array<[number, number]>): number[] {
|
||||
const byParent = new Map<number, number[]>()
|
||||
for (const [pid, ppid] of rows) {
|
||||
const children = byParent.get(ppid) ?? []
|
||||
children.push(pid)
|
||||
byParent.set(ppid, children)
|
||||
}
|
||||
const result: number[] = []
|
||||
const queue = byParent.get(root) ?? []
|
||||
for (let index = 0; index < queue.length; index += 1) {
|
||||
const pid = queue[index]
|
||||
if (pid === undefined) continue
|
||||
result.push(pid)
|
||||
queue.push(...(byParent.get(pid) ?? []))
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Format every independently observed failure fact for the aggregate summary.
|
||||
* @param result - unsuccessful gate result.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue