Reviewer walkthrough · NVIDIA/OpenShell#4320
Deliver gateway configuration over the supervisor session
Supervisors poll the gateway every 10 seconds today. This PR lets the gateway push configuration over the ConnectSupervisor stream instead. Supervisors apply each update and acknowledge it. You turn it on with config_delivery_mode = "push". Poll stays the default.
For users
Nothing changes unless an operator opts in. With push on, a policy or settings change reaches the sandbox without waiting for the next poll.
For reviewers
Start with the session protocol and the per-session delivery task. Most of the correctness argument lives there. The rest is plumbing, moved code, and generated bindings.
Getting around
Use ← → or the buttons at the bottom. Slides tagged interactive have controls.
High level
How a config change reaches a running sandbox
Same boxes, two triggers. In poll mode the supervisor asks on a timer, and the gateway builds a snapshot per request. In push mode the write itself starts the build, and the gateway sends the result down the session stream. Both modes apply through the same supervisor code.
Before poll, default
After push, opt-in
Highlighted boxes differ between the two modes. In push mode every update carries the sandbox configuration and the provider environment it names, and the gateway resolves credentials only when that name changes. In poll mode the supervisor fetches the provider environment only when its name changes.
High level
Terms used in the code
These names show up in code, metrics, and logs.
| Term | Meaning |
|---|---|
| Update | One ConfigUpdate: the sandbox_config (policy, settings, middleware, admission) and the provider_environment it names (env values, credentials, managed files). The gateway always sends a full snapshot of both, never a diff. |
| Bootstrap | The session's first update, sent inside SessionAccepted. The supervisor has already started from polled configuration, as on main. It applies the bootstrap as an ordinary update and stops polling. |
| Dirty mark | When a write commits, the gateway marks every affected session dirty. The mark is one flag and carries no data, so more writes merge into it. The code calls the write a publication. |
| Build permit | Limits concurrent builds. One FIFO pool, sized from the database pool. |
| Delivery task | One tokio task per push session. It waits for a dirty mark, takes a permit, builds, and hands the update to the session. |
| In flight | The update the supervisor hasn't answered yet. A session has at most one. A newer build waits in one pending slot, and the newest build wins. |
| Result | ConfigUpdateResult from the supervisor: the outcome, the admission, and the policy load result when the update changed the sandbox policy. |
| Reconcile | Every 30 s each session rebuilds its update anyway, to repair anything a missed mark left stale and to resend an update the supervisor rejected. |
Mechanism · config_delivery/session.rs, supervisor_session.rs interactive
Session protocol, from hello to live updates
Supervisors start exactly as on main: they poll for their startup configuration, start the workload, then open the session. A push session hands them a bootstrap, after which the supervisor stops polling and each update carries the whole configuration, one at a time. Click a message or step through.
Mechanism · config_delivery/mod.rs, task.rs
Mark dirty, wait for a permit, build, send
Marking a session dirty records intent only. The task reads state after it holds a permit, so each build sees every write that marked it. That is why merging marks is safe.
Who marks what
| Write | Scope |
|---|---|
| Sandbox policy or settings, provider attach or detach | one sandbox |
| Provider record change, credential refresh | sandboxes attaching it |
| Provider profile change | workspace or all |
| Global policy or settings, interceptor profile revision | all connected |
| 30 s reconcile | itself |
Every mark rebuilds the sandbox configuration. The provider environment is reused unless the configuration names a different one: a new provider revision, attachment epoch, or policy hash. Only then does the build resolve credentials.
Queueing
- Permits are FIFO. Tokio's
Semaphoreis fair, so 100 waiting sessions get permits in the order they asked. - More marks don't wake a waiting task, so it keeps its place in line.
- Every write and every reconcile waits in the same line. There are no priority lanes, so an edit to one sandbox can wait behind a fleet-wide change that is already queued.
- A build that finishes while an update is in flight waits in the session's pending slot. A newer build replaces it.
task.rs · acquire_permit
Permit sizing
Builds = 2 × DB pool size, at least 4.
| Store, default pool | Pool | Builds |
|---|---|---|
| Postgres | 10 | 20 |
| SQLite file | 5 | 10 |
| SQLite memory | 1 | 4 |
Why the DB pool? Each build is a burst of small store queries, so the pool is the shared bottleneck. Two builds per connection keep it busy without piling waiters onto the pool's acquire timeout. The pool follows database_max_connections, and the build limit follows the pool. Credential resolution also waits on drivers like Vault, and this sizing ignores that.
Mechanism · task.rs interactive
Watch marks merge and updates wait their turn
Eight push sessions on a gateway with the smallest sizing, 4 build permits. Time runs slow so you can follow it. Try Global setting: every session rebuilds, but none resolves credentials. Then try 5 quick edits on sb-3: five writes become one build.
Simplified model. One tick is 0.5 s. Input load takes 1 tick, the sandbox-config build 1, resolving credentials 2 more, and the supervisor answers after 3. Reconcile runs every 30 s, starting 30 to 60 s in. HA, owner checks, and the bootstrap are left out.
Mechanism · task::run, RECONCILE_INTERVAL interactive
Reconcile catches what dirty marks miss
Dirty marks live in memory, and peer notifications can fail. So every 30 s each session rebuilds its update. An update the supervisor already applied is not sent again. One it rejected is sent again. The first pass lands at a random point 30 to 60 s after the session registers.
Green is the bootstrap at connect, blue is a reconcile build. The bars count builds per 5 s across 40 sessions. Without the jitter, a gateway restart would line every session up on the same tick for good.
What a reconcile costs
Resolving credentials may call Vault, so a reconcile avoids it unless something changed.
- Build
sandbox_configfrom the store. - If it still names the session's cached provider environment (same revision, attachment epoch, and policy hash), reuse it. Otherwise resolve credentials.
- If the supervisor already applied this exact update, send nothing.
Polling supervisors follow the same rule: they fetch the provider environment only when its name changes. If credentials can't be resolved, the update carries an empty environment marked as failed. The supervisor revokes its static credentials and rejects the update, and the gateway retries the build with backoff.
Mechanism · timeouts and retries
Every wait has a bound
Delivery is best effort and runs beside the session. A failed build never tears a session down. It retries, and reconcile covers whatever retries miss.
Timeouts
| Sandbox-config build | 5 s |
| Provider-env build | 20 s |
| Bootstrap build | 45 s |
| Supervisor answering the bootstrap | 2 min |
| Peer notify | 5 s |
Loading the inputs counts against the sandbox-config deadline. An unanswered update has no timeout: the supervisor answers every update in order, so a session that never answers is gone.
Build failure backoff
Per session, 1, 2, 4, 8, 16, then 30 s. A new write or a session close cuts the wait short. A success resets it.
A failed credential resolution counts as a failed build, so it backs off the same way.
Supervisor-side failures
- The gateway counts an update as delivered only when the supervisor applied it and its results are persisted. Reconcile resends anything else.
- A bad bootstrap result closes the session. The supervisor reconnects with
ReconnectBackoff. - A result that answers no update in flight, or a session stream that stops draining, also closes the session. The supervisor reconnects and gets a fresh bootstrap.
- Failure messages carry no payloads, credentials, or provider values.
- Startup is unchanged from main, including a degraded start when middleware is unreachable.
- A failed session setup demotes the sandbox. The shutdown handoff from #4321 still applies.
Mechanism · PeerNotifyConfigUpdate
Several replicas: tell the owner, send no secrets
A write can land on any replica. Only the replica that owns a sandbox's session can deliver to it. So the writing replica sends the owner a notification with no config in it, and the owner rebuilds from the shared store.
Merged and bounded
Notifications to the same target merge while one is pending or running. At most 8 run at once, each with a 5 s timeout.
Tied to one session
A sandbox notification carries the session_id the sender saw. If the session has moved, the receiver answers stale_owner. A build that finishes after the owner moved notifies the new owner.
Skipped on one replica
is_single_replica() skips the owner lookup and peer notifications entirely. Workspace, provider, and global changes go to every peer.
Also helps poll mode
Builds got cheaper in both modes
Push builds when a write commits, not when a supervisor asks, so a fleet-wide change means a burst of builds. These changes cut the cost of each one. Most help poll-only gateways too.
One input load per build
The provider catalog, provider records, global settings, and latest sandbox policy load once into SandboxConfigInputs and feed both parts of an update. The bootstrap, live builds, and the readiness snapshot all use it.
grpc/policy/config_inputs.rs
Cached provider profiles
Every build used to call the interceptor provider-profile sources. In push mode the gateway serves them from a snapshot it refreshes every 10 s. The interval is fixed. When the revision changes, all connected sessions get marked dirty. Poll mode still calls the sources on every read.
Reads fail closed before the first fetch and after 30 s of failed refreshes.
provider_profile_sources.rs
Provider environment builds
They read refresh state once instead of twice and load providers concurrently. Only these builds resolve credentials, and a push session reuses its last result until its configuration names a different environment.
grpc/provider.rs, provider_refresh.rs
Rollout interactive
Opt-in, safe across versions, easy to undo
Push needs a gateway in push mode and a supervisor that says it supports it. Every other pairing polls exactly as today. Pick one.
Rolling back
Set config_delivery_mode = "poll" and restart the gateway. Supervisors reconnect, get config_push_enabled = false, and go back to polling. There is no migration. Poll mode reads no state that this PR adds.
Mixed replicas
An older replica doesn't implement PeerNotifyConfigUpdate, and senders count that as unsupported_peer. Its sessions poll anyway, so nothing is lost. Push sessions on newer replicas still reconcile every 30 s.
Ship it off
Default is poll. New supervisors advertise support. CI runs both e2e:rust and e2e:rust:push.
Opt in
Turn push on for dev and internal gateways. Watch queue wait, build failures, apply results, and bootstraps. Load test push against poll, then HA with 3 replicas.
Default to push
Push becomes the default and the only mode, and supervisors start over the session. Needs a fresh deployment.
Remove polling
Drop GetSandboxConfig and GetSandboxProviderEnvironment in the same release. The proto comments already call them polling projections.
Phases 2 and 3 are tracked in #4498. They are not part of this PR.
Low level
Where to spend review time
Generated Go bindings and moved supervisor code make up most of the line count. The new gateway logic is about 1.9k lines in config_delivery/, not counting tests, plus the handshake in supervisor_session.rs. About 60% of session.rs is tests.
| File | + | − | Read for |
|---|
Suggested order
proto/openshell.proto, the contract.config_delivery/task.rs. Marks, the build permit, reconcile, backoff. Short and well tested.config_delivery/mod.rs. Publishing, registration, builds, the provider environment cache, owner check, peer notify.config_delivery/session.rs. Bootstrap, the update in flight, results.supervisor_session.rson the server, for the handshake.supervisor-process/supervisor_session.rsfor the switch from polling to push, thenconfig_runtime.rs. Usegit diff --color-movedfor the latter.
Open questions
- Is FIFO fair enough across workspaces, or do we need round-robin now? One busy workspace can fill the queue.
- Without priority lanes, an edit to one sandbox waits behind a fleet-wide change already queued. Is that acceptable at our fleet sizes?
- Should the build limit be configurable on its own? Today it follows
database_max_connectionsand ignores credential driver latency. - Is 30 s the right reconcile interval for large fleets? It's a constant.
- Should provider readiness reports move onto the supervisor stream after this lands?
- Durable config update operations are deferred. #1731's acceptance criteria need an amendment.