fix(server): prevent exec relays from hanging on idle connections

[Gal-Zaidman]

Summary

Gateway ExecSandbox calls could hang indefinitely after the command finished, when a supervisor session reset mid-exec orphaned the relay channel — the exec loop blocked on channel.wait() with no liveness backstop, so callers hung until their own deadline. This adds SSH and HTTP/2 keepalives and bounds the post-exit wait so a wedged/orphaned relay fails fast instead of hanging.

Related Issue

Closes #1990

Changes

• Exec relay russh client: SSH keepalive so a wedged/orphaned relay is shed (returns an error) instead of parking on channel.wait() forever. Channel-silent execs (e.g. an agent that redirects stdout to a file) stay alive while the relay is healthy — liveness is probed via keepalive, not output-idle.
• Bound how long the gateway waits for the trailing channel close after a command reports its exit status.
• Return UNAVAILABLE when a relay closes before reporting an exit status, instead of a misleading exit code 1.
• Server-side HTTP/2 keepalive (with the required Timer) on supervisor multiplex connections, to reduce the session resets that orphan relays.
• Documented the relay-liveness backstops in architecture/gateway.md.

Testing

• mise run pre-commit — rust:format:check, cargo clippy -D warnings, and markdownlint are clean for this change (ran individually). Note: the local mise run pre-commit aborts on its python:proto step due to a missing grpc_tools dev dependency in the venv, unrelated to this change; CI runs the full suite.
• Unit tests added/updated — none; the exec loop requires a live SSH relay, so there is no isolated unit surface.
• E2E tests added/updated — N/A; reproducing the fix path needs a mid-exec supervisor-session reset, which isn't reliably stageable in an in-process e2e.

Manually validated on a Kubernetes deployment: rebuilt and deployed the gateway image, confirmed the gateway is healthy, that long channel-silent execs are not killed by the keepalive, and that the previously-observed multi-sandbox hang no longer reproduces.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

[Gal-Zaidman]

I have read the DCO document and I hereby sign the DCO.

[Gal-Zaidman]

recheck

[TaylorMutch]

/ok to test b4878be

[TaylorMutch]

@Gal-Zaidman have you been able to verify this resolves the issue in your environment?

[Gal-Zaidman]

@Gal-Zaidman have you been able to verify this resolves the issue in your environment?

Yes, currently ran a job with 80 concurrent agents each running an SWE bench task with long exec (that is how harbor works) - zero hangs.
Before the fix, even with 20 agents, more than half would have hang.

[TaylorMutch]

/ok to test b4878be

[github-actions]

Label test:e2e applied for b4878be. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.