Chaperon: Secure Slurm Proxy for Sandboxed Sessions¶

Disclaimer: The chaperon is a best-effort security mechanism. It has not been formally audited and comes with no guarantees. While it aims to prevent sandbox bypass via Slurm, there may be edge cases, Slurm version-specific behaviors, or site-specific configurations that weaken its protections. Review the Security Properties and Known Limitations sections, and test against your environment before relying on it. Use at your own risk.

Problem¶

The previous Slurm isolation model exposed the munge authentication socket (/run/munge) read-only inside the sandbox, allowing the PATH-shadowing wrappers to authenticate with slurmctld. This meant anything inside the sandbox that could talk to munge — a crafted binary, Python ctypes, a direct socat call — could submit jobs without going through the sandbox wrappers. PATH shadowing was a speed bump, not a wall.

The chaperon replaces this with a zero-trust architecture: all Slurm authentication assets are blocked inside the sandbox, and a proxy process running outside the sandbox validates, wraps, and submits jobs on behalf of the sandboxed process.

User-Facing Experience¶

Although every Slurm call from inside the sandbox is heavily filtered by the chaperon (argument whitelisting, CWD validation, scope-filtered output, denied subcommands, comment-tag injection and stripping), interacting with Slurm through any utility from within the sandbox is indistinguishable from interacting with it from outside the sandbox. The stubs shadow the real binaries on PATH, the exit codes match, and stdout/stderr are passed through verbatim (with only chaperon-internal metadata stripped). An agent running sbatch, srun, squeue, scancel, scontrol show job, sacct, sinfo, etc. sees output that looks like an unmodified Slurm CLI. The filtering is transparent in the allowed case, and explicit in the denied case: rejected flags and blocked subcommands fail with a clear error message instead of a silent rewrite.

Architecture¶

sandbox-exec.sh
  │
  ├── mktemp -d → _CHAPERON_FIFO_DIR (chmod 700)
  ├── mkfifo req (chmod 600)
  ├── backend_prepare (bind-mounts FIFO dir into sandbox)
  ├── fork chaperon (reads req pipe, outside sandbox)
  ├── close FDs 3+ (no exceptions needed — FIFOs are filesystem-based)
  ├── export _CHAPERON_FIFO_DIR
  └── backend_exec → enters sandbox
        │
        └── Inside sandbox:
              - /run/munge BLOCKED (no munge auth)
              - /usr/bin/{sbatch,srun,...} BLOCKED
              - /etc/slurm/ BLOCKED
              - stub sbatch → writes to req FIFO (flock for atomicity)
                → chaperon validates, closes FD 3 for handler children
                (3>&-), wraps in sandbox-exec.sh, calls real sbatch
                → writes response to per-request FIFO → stub reads result
              - stub srun → writes to req FIFO → chaperon validates flags:
                  alloc mode (login node): wraps command in sandbox-exec.sh,
                    calls real srun → compute node runs sandboxed
                  step mode (compute node, SLURM_JOB_ID set): execs real
                    srun directly for job-step launching (MPI)
              - stub scancel → writes to req FIFO → chaperon validates
                job scope, calls real scancel

Component Roles¶

Trusted side (runs outside the sandbox, has Slurm credentials):

Component	Role
chaperon.sh	Main loop: reads requests from FIFO, dispatches to handlers, writes responses
handlers/_handler_lib.sh	Shared utilities: argument whitelisting, CWD validation, job wrapping, comment tag encoding/stripping
handlers/sbatch.sh	Validates args, wraps job in `sandbox-exec.sh`, submits to real sbatch
handlers/srun.sh	Validates flags; allocation mode wraps in `sandbox-exec.sh`, step mode execs real srun
handlers/scancel.sh	Validates job scope, forwards to real scancel
handlers/squeue.sh	Scopes output to session/project, strips chaperon tags
handlers/scontrol.sh	Scoped `show job`, `hold`, `release`, `requeue`, `update`; strips chaperon tags
handlers/sacct.sh	User-scoped accounting; strips chaperon tags
handlers/sacctmgr.sh	Read-only cluster/QOS/TRES queries; blocks user enumeration
handlers/sinfo.sh, sstat.sh, sprio.sh, sshare.sh, sdiag.sh	Read-only or user-scoped passthrough
handlers/sreport.sh, blocked.sh	Blocked (user enumeration risk / unsupported commands)

Untrusted side (runs inside the sandbox, no Slurm credentials):

Component	Role
stubs/{sbatch,srun,scancel,...}	PATH-shadow the real Slurm binaries; serialize the user's command into a FIFO request and relay the response
stubs/_stub_lib.sh	Shared stub helpers: FIFO communication, request framing, response parsing

Shared:

Component	Role
protocol.sh	`CHAPERON/1` wire protocol: base64 encode/decode, message framing

File Structure¶

chaperon/
├── chaperon.sh              # Main loop (runs OUTSIDE sandbox)
├── protocol.sh              # Read/write protocol messages (shared)
├── handlers/
│   ├── _handler_lib.sh      # Arg whitelisting, CWD validation, job wrapping
│   ├── sbatch.sh            # Validates, wraps, submits via real sbatch
│   ├── srun.sh              # Validates srun flags, wraps or execs real srun
│   ├── scancel.sh           # Validates job scope, cancels via real scancel
│   ├── squeue.sh            # Filters squeue output to scoped jobs
│   ├── scontrol.sh          # Scoped scontrol: show, hold, release, update
│   ├── sacct.sh             # User-scoped sacct (--allusers denied)
│   ├── sacctmgr.sh          # Read-only cluster/QOS/TRES queries
│   ├── sinfo.sh             # Read-only partition/node info
│   ├── sstat.sh             # User-scoped job step statistics
│   ├── sprio.sh             # User-scoped job priority factors
│   ├── sshare.sh            # User-scoped fairshare data
│   ├── sdiag.sh             # Read-only scheduler diagnostics
│   ├── sreport.sh           # Blocked (user enumeration risk)
│   └── blocked.sh           # Generic "command blocked" response
└── stubs/                   # PATH-shadowing stubs (all talk to chaperon)
    ├── _stub_lib.sh         # Stub→chaperon communication library
    ├── sbatch, srun, scancel, squeue, scontrol
    ├── sacct, sacctmgr, sinfo, sstat, sprio, sshare, sdiag, sreport
    └── salloc, sattach, sbcast, scrontab, scrun, strigger  # blocked

Protocol: `CHAPERON/1`¶

Line-based, all user data base64-encoded (injection-proof in bash).

Request (stub → chaperon)¶

CHAPERON/1 sbatch
ARG <base64>          # one per sbatch flag/value
ARG <base64>
CWD <base64>          # working directory (validated by handler)
SCRIPT <base64>       # job script content (--wrap converted to script)
RESP_FIFO <path>      # raw filesystem path (not base64) — validated by chaperon
END

Note: protocol.sh provides chaperon_send_request() as a helper, but _stub_lib.sh builds the request message directly (to include the RESP_FIFO field and support atomic writes via flock). The chaperon main loop also parses requests inline rather than using chaperon_read_request(), in order to support read timeouts and parent liveness checks.

Response (chaperon → stub)¶

CHAPERON/1 RESULT
EXIT <number>         # exit code from real sbatch
STDOUT <base64>       # stdout (e.g., "Submitted batch job 12345")
STDERR <base64>       # stderr (e.g., validation errors)
END

Safety Properties¶

base64 -w 0: single-line encoding prevents newline injection
IFS= read -r: prevents word splitting on read
Unknown lines silently ignored: forward compatibility — new fields can be added without breaking old readers
No shell interpretation: the chaperon never passes user data to sh -c, eval, or any form of shell expansion

FIFO Design¶

The communication channel uses named pipes (FIFOs) in a per-session temporary directory created before the sandbox is entered:

Per-session directory: mktemp -d creates a directory with chmod 700 — only the owning user can access it. The directory is bind-mounted into the sandbox so both sides can reach it.
Persistent request pipe: A single req FIFO handles all requests. The chaperon opens it O_RDWR to prevent blocking and avoid EOF between requests.
Per-request response pipes: Each stub creates an atomically-named directory (mktemp -d) containing a response FIFO (fifo), sends the path in the request, and reads the response from it. This eliminates the TOCTOU window that mktemp -u + mkfifo would have. The FIFO is created with mkfifo -m 600 (permissions set atomically).
No FD inheritance needed: Unlike socketpairs, FIFOs are filesystem-backed and survive bwrap's FD closing (which closes all FDs > 2). No exemptions needed.
Timeouts: The stub reads responses with a 30-second timeout (chaperon_read_response in _stub_lib.sh) to prevent infinite hangs if the chaperon dies. The chaperon uses a 30-second body read timeout and a 10-second response write timeout to prevent stalls from malicious or dead stubs. All internal squeue calls (for scope resolution) use timeout 10.
Cleanup on exit: The chaperon's EXIT trap removes the entire FIFO directory.

Chaperon Lifecycle¶

Creation: sandbox-exec.sh creates a FIFO directory via mktemp -d and a request pipe via mkfifo, launches chaperon.sh as a background process, and exports _CHAPERON_FIFO_DIR for the sandbox.
Logging: The chaperon initializes structured logging via logging.sh (see Logging below). The log file is created before the main loop starts.
Orphan prevention: The chaperon sets PR_SET_PDEATHSIG via Python/ctypes so it receives SIGTERM if its parent (sandbox-exec.sh) dies. This prevents orphaned chaperon processes.
Signal handling: SIGTERM and SIGINT are trapped for clean shutdown (FD cleanup, shutdown log entry).
Main loop: Reads requests inline with timeouts (30-second body timeout to prevent stalls from malformed requests), dispatches to the appropriate handler with FD 3 closed (3>&-) to prevent child processes from inheriting the request FIFO, captures stdout/stderr, and writes the response via the held response FD with a 10-second write timeout. Each request is logged with its full arguments, and handler stderr (including _sandbox_deny / _sandbox_warn messages) is captured in the log.
Exit: On read error, parent death (liveness polling), or signal, the chaperon logs shutdown, removes the FIFO directory, and exits 0.

Note: protocol.sh provides a chaperon_read_request() helper, but the main loop in chaperon.sh performs its own inline parsing to support read timeouts and liveness checks that the helper does not provide.

Logging¶

The chaperon writes structured, timestamped logs to per-session files for debugging and security auditing.

Log Location¶

${XDG_STATE_HOME:-~/.local/state}/agent-sandbox/chaperon/<hostname>_<PID>_<timestamp>.log

Filenames include the hostname so that multiple sandboxes across machines sharing an NFS home directory produce distinct, non-conflicting log files. Each chaperon instance gets its own file.

Log Format¶

Each log file starts with a metadata header (host, PID, PPID, project dir, slurm scope, log level), followed by timestamped structured lines:

2026-04-15T20:34:49Z [INFO] starting (pid=12345, ppid=12340, fifo=/tmp/chaperon-abc123)
2026-04-15T20:34:50Z [INFO] request: sbatch args=[--partition=campus-new --mem=4G] cwd=/fh/fast/lab/project
2026-04-15T20:34:50Z [INFO] request: sbatch script=247 bytes
2026-04-15T20:34:50Z [DEBUG] handler sbatch exited 0
2026-04-15T20:34:52Z [INFO] request: salloc args=[--partition=campus] cwd=/fh/fast/lab/project
2026-04-15T20:34:52Z [WARN] handler salloc exited 1
2026-04-15T20:34:52Z [WARN] handler salloc stderr: sandbox: 'salloc' is not allowed inside the sandbox.
2026-04-15T20:34:52Z [WARN] handler salloc stderr:   ⚠ This action was blocked for security. [...]
2026-04-15T20:35:00Z [INFO] shutting down (pid=12345)

What is Logged¶

Level	Content
ERROR	Security rejections (RESP_FIFO validation failures, symlink attacks, invalid command names)
WARN	Handler non-zero exits, handler stderr output (`_sandbox_deny` / `_sandbox_warn` messages), response write timeouts, missing RESP_FIFO
INFO	Startup/shutdown, each request with full arguments and CWD, script size and shebang
DEBUG	Handler success exit codes

Handler stderr is particularly important for security audit: it captures every _sandbox_deny (bypass attempts, scope violations) and _sandbox_warn (unrecognized flags, usage errors) with the full denial message.

Script body content is intentionally not logged at any level. Scripts may contain embedded secrets (API keys, database credentials), PHI/PII, or paths to restricted data. The command arguments, CWD, script size, shebang, and handler denial messages provide a sufficient audit trail without the secret exposure risk. Log files are restricted to owner-only access (chmod 700 on the directory, chmod 600 on each file) as defense-in-depth.

Configuration¶

Two settings in sandbox.conf (or per-project conf.d/*.conf):

# Log level: debug, info (default), warn, error
CHAPERON_LOG_LEVEL="info"

# Days to retain log files (pruned at each chaperon startup)
CHAPERON_LOG_RETAIN_DAYS=7

Retention¶

At startup, the chaperon prunes old logs in the background (non-blocking):

Age-based: Log files older than CHAPERON_LOG_RETAIN_DAYS are deleted.
Size-based: If total log size exceeds 50 MiB, the oldest files are removed first (the current session's log is never deleted).

This prevents unbounded accumulation on shared NFS home directories even with many concurrent sandboxes across machines.

Debugging a Dead Chaperon¶

When a chaperon dies unexpectedly, the log file persists (it is not in the ephemeral FIFO directory). To investigate:

# Find recent chaperon logs
ls -lt ~/.local/state/agent-sandbox/chaperon/

# Check the last entries of a specific session
tail -20 ~/.local/state/agent-sandbox/chaperon/<hostname>_<pid>_<timestamp>.log

# Find all security denials across sessions
grep '\[WARN\].*stderr:.*sandbox:' ~/.local/state/agent-sandbox/chaperon/*.log

# Find all denied commands
grep '\[WARN\].*stderr:.*⚠' ~/.local/state/agent-sandbox/chaperon/*.log

For deeper tracing, set CHAPERON_LOG_LEVEL=debug to capture handler success exit codes and submitted script content.

Handler Dispatch¶

Filesystem-based: when a request for command X arrives, the chaperon looks for handlers/X.sh and calls handle_X(). If no handler exists, handlers/blocked.sh is used.

This design makes it trivial to add support for new commands (drop a handler file) or block them (they're blocked by default).

sbatch Handler¶

The sbatch handler (handlers/sbatch.sh) performs four validation/transformation steps before submission:

CWD validation: The requested working directory must be a physical path under the project directory (resolves symlinks to prevent escape). Both sbatch and srun (allocation mode) validate CWD.
Argument whitelisting: Every flag is checked against _SBATCH_ALLOWED_FLAGS (~40 safe flags). Denied flags cause immediate rejection with a clear error message.
--output / --error path transformation (bwrap/firejail only): values are redirected to $project_dir/.sandbox-state/slurm-logs/; the in-sandbox wrapper creates a relative symlink from the user's intended path to the staging file. Closes the slurmstepd-side O_NOFOLLOW-absence escape (e.g., sbatch --output=/etc/cron.d/evil). Disabled on landlock — see docs/reference/chaperon-output-staging.md.
Job wrapping: The user's script is written to a temp file, and a wrapper script is generated that runs it inside sandbox-exec.sh --project-dir $PROJECT_DIR. The wrapper is submitted to the real sbatch.

Job Tagging and Scoping via `--comment`¶

The chaperon needs to track which jobs belong to which sandbox session/project so that squeue, scancel, and other scoped handlers can filter appropriately. This is done by injecting a structured tag into Slurm's --comment field on every submitted job.

Why --comment and not --job-name? The --job-name (-J) field is user-visible, commonly set by workflows and scripts, and used for human identification (e.g., alignment-step1, train-model). Overwriting or prefixing it would break existing workflows that parse job names. The --comment field, by contrast, is rarely used in practice — it's a free-text metadata field that most users never set, and it doesn't appear in default squeue output. This makes it ideal for machine-readable tagging without interfering with user workflows.

What happens to user-supplied comments? If the user passes --comment "my note" to sbatch, the chaperon preserves it by appending it (percent-encoded) to the tag as user=my%20note. The encoding prevents crafted comments from injecting fake chaperon:, sid=, or proj= patterns that would confuse scope filtering. When the sandbox user queries jobs (via squeue, scontrol show job, or sacct), the chaperon tag is automatically stripped and the original comment is restored — the user sees my note, not the internal tag.

Tag format¶

chaperon:sid=<session_id>,proj=<project_hash>[,user=<original_comment>]:END

Field	Content	Purpose
`sid`	`<PID>.<epoch>`	Unique per chaperon instance (session scope). Set once at startup and guarded against re-initialization when `_handler_lib.sh` is re-sourced per handler dispatch.
`proj`	First 12 hex of `md5(project_dir)`	Groups jobs by project (project scope)
`user`	User's original `--comment` value (percent-encoded)	Preserves user metadata
`:END`	Literal end marker	Unambiguous tag boundary for stripping (colons are percent-encoded in user values, so `:END` cannot appear inside the encoded comment)

This tag is set once at submission and is inherited by array tasks, survives preemption (job ID may change, comment does not), and is queryable via squeue/sacct:

squeue --me -h -o "%i %k" | grep "chaperon:sid=$SID"     # session
squeue --me -h -o "%i %k" | grep "chaperon:.*proj=$HASH"  # project
squeue --me -h -o "%i %k" | grep "chaperon:"              # all sandbox jobs

scancel Handler¶

The scancel handler (handlers/scancel.sh) queries squeue --comment to resolve which jobs are in scope, then passes only matching IDs to the real scancel. No file-based tracking — the tag in Slurm is the source of truth.

Argument whitelisting: Only safe scancel flags are forwarded. Flags like --user, --me, --account, --wckey are denied — scope is controlled by the chaperon.
Job ID validation: Positional arguments must be numeric job IDs.
Scope filtering: Requested job IDs are checked against squeue output filtered by the chaperon tag.
scancel all: Cancels everything within scope (no specific IDs needed).

Scope Levels¶

Configured via SLURM_SCOPE in sandbox.conf. Applies to scancel, squeue, scontrol, and sstat:

Scope	Behavior	Filter
`session`	Only jobs from THIS sandbox session	`chaperon:sid=<this_session>`
`project` (default)	Jobs from any sandbox with same project dir	`chaperon:.*proj=<hash>`
`user`	All jobs of the current user (including non-sandbox jobs)	`squeue --me`
`none`	No scope restriction (full access to your own jobs)	`squeue --me`

Denied sbatch Flags¶

These flags are explicitly rejected because they could bypass sandboxing:

Flag	Reason
`--wrap`	Reconstructed by the handler (user data from protocol, not shell)
`--chdir` / `-D`	CWD comes from protocol and is validated against project dir
`--uid` / `--gid`	Must not impersonate other users
`--get-user-env`	Can leak host environment variables
`--propagate`	Can propagate unsafe resource limits
`--prolog` / `--epilog`	Run arbitrary scripts outside sandbox control
`--task-prolog` / `--task-epilog`	Same as above
`--burst-buffer-file` / `--bbf`	Arbitrary file access
`--bcast`	Copy binary to compute nodes (bypass wrapping)
`--container`	OCI container execution could bypass sandbox wrapping

Unknown flags (not in the whitelist) are also rejected.

srun Handler¶

The srun handler (handlers/srun.sh) operates in two modes:

Allocation mode (no SLURM_JOB_ID — login node): 1. Validates CWD is under the project directory (same check as sbatch) 2. Validates flags against a whitelist that includes scheduling flags (-p, -A, -t, etc.) 3. Wraps the command in sandbox-exec.sh --project-dir $DIR so the compute-node process inherits sandbox restrictions 4. Calls real srun with the validated flags and wrapped command

Step mode (SLURM_JOB_ID set — inside a compute-node allocation): 1. Validates flags against a step-only whitelist (no scheduling flags — steps inherit the job's resources) 2. Execs real srun directly — the command runs within the existing sandboxed allocation

Denied srun flags (both modes):

Flag	Reason
`--pty`	No PTY passthrough via the chaperon protocol
`--jobid` / `-j`	Cannot attach to arbitrary allocations
`--uid` / `--gid`	Must not impersonate other users
`--chdir` / `-D`	CWD comes from protocol and is validated
`--get-user-env`	Can leak host environment variables
`--propagate`	Can propagate unsafe resource limits
`--prolog` / `--epilog` / `--task-prolog` / `--task-epilog`	Arbitrary script execution
`--bcast`	Binary broadcast (bypass wrapping)
`--container`	OCI container execution bypasses sandbox
`--network`	Network namespace manipulation

In step mode, allocation flags (-p, -A, -t, -q, --reservation, etc.) are also denied.

squeue Handler¶

The squeue handler (handlers/squeue.sh) filters queue output to only show jobs within scope. Uses the SLURM_SCOPE setting from sandbox.conf.

Flags like --user, --me, --account are denied (scope controlled by chaperon)
If specific job IDs are requested via -j, they're validated against scope
Otherwise, all jobs in scope are shown
All internal squeue calls use timeout 10 to prevent hangs if slurmctld is unresponsive

scontrol Handler¶

The scontrol handler (handlers/scontrol.sh) allows a subset of scontrol subcommands with scope enforcement:

Subcommand	Scoped?	Notes
`show job [ID]`	Yes	Shows only chaperon-submitted jobs
`show node/partition/config/step`	No	Read-only system info
`hold JOBID`	Yes	Must be in scope
`release JOBID`	Yes	Must be in scope
`requeue JOBID`	Yes	Must be in scope
`update job JOBID Key=Val`	Yes	Must be in scope; only safe update keys allowed

Denied subcommands: shutdown, reconfigure, create, delete, and all others.

Denied update keys: UserId, GroupId, WorkDir, AdminComment, and all keys not in the whitelist. See handlers/scontrol.sh for the full list.

sacct Handler¶

The sacct handler (handlers/sacct.sh) enforces user-level scoping:

Always injects --user=$(whoami) — only the current user's jobs are shown
--allusers and --accounts are denied
--user / --uid / --me: self-scoped values (--user $USER, --user=$USER, --me, --uid $(id -u)) are silently accepted — they're equivalent to the auto-inject. Cross-user values are denied with an actionable message ("drop the flag, or pass --me")
Job-level scoping (by chaperon comment) is intentionally not applied — sacct is retrospective and the full job history is useful for debugging

sacctmgr Handler¶

The sacctmgr handler (handlers/sacctmgr.sh) is heavily restricted to prevent user/group enumeration:

Subcommand	Allowed targets
`show` / `list`	`cluster`, `qos`, `tres`, `configuration`

Denied show targets: user, account, association, coordinator, event, reservation, transaction, wckey — all expose user/group data.

Denied subcommands: add, modify, delete, archive, dump, load, reconfigure — all write operations.

sinfo Handler¶

Read-only passthrough for partition/node status. No scoping needed — this is system information. Unknown flags are rejected.

sstat Handler¶

Shows statistics for running job steps. Job step IDs (format jobid.stepid) are validated — the base job ID must be in the current scope. Uses _validate_job_in_scope from _handler_lib.sh.

sprio Handler¶

Shows priority factors for pending jobs. Always injects --user=$(whoami) to scope output to the current user. --allusers and --user are denied.

sshare Handler¶

Shows fairshare data. Always injects --user=$(whoami). Denies --all, --user, and --accounts to prevent enumerating other users' fairshare allocations.

sdiag Handler¶

Read-only scheduler diagnostics passthrough. Denies --reset (clears scheduler statistics — a write operation).

sreport Handler¶

Blocked entirely. sreport generates accounting reports across many sub-report types, many of which enumerate users and accounts. Use sacct with formatting options for similar data scoped to your user.

Comment Stripping¶

The chaperon injects structured tags into Slurm's --comment field for scoping, but the sandbox user should never see these internals. The _strip_chaperon_tags() function in _handler_lib.sh is piped over the output of all user-facing handlers that may display comments: squeue, scontrol (show job), and sacct.

The stripping pipeline: 1. Extracts the user value: regex matches the full tag chaperon:sid=...,proj=...[,user=VALUE]:END and replaces it with just VALUE (or empty string if no user comment was set) 2. Decodes percent-encoding: restores %2C → ,, %3A → :, %3D → =

The :END marker makes extraction reliable across all Slurm output formats (tabular, parsable/pipe-delimited, JSON, YAML, scontrol key=value). Because colons are percent-encoded (%3A) in user values, :END cannot appear inside the encoded comment, providing an unambiguous boundary.

Result: sandbox users see their original --comment value (or an empty field) — the Slurm interface behaves as expected with no chaperon artifacts visible.

Comments on jobs not submitted through the chaperon (e.g., in user/none scope modes) pass through unchanged.

Blocked Commands¶

The following commands are routed to blocked.sh (no handler):

Command	Reason
`salloc`	Interactive allocations can't be safely proxied (requires PTY)
`sattach`	Attaching to step I/O could interfere with other jobs
`sbcast`	File broadcast bypasses sandbox restrictions
`scrontab`	Recurring jobs would bypass sandbox wrapping
`scrun`	OCI container launcher bypasses sandbox
`strigger`	Event triggers can execute arbitrary commands

What Gets Blocked Inside the Sandbox¶

Resource	bwrap	firejail	landlock
`/run/munge/` (auth socket)	Hidden (tmpfs /run, not re-mounted)	`--blacklist=/run/munge`	NOT BLOCKED — Landlock cannot restrict `AF_UNIX connect()`
`/usr/bin/{sbatch,srun,...}`	`--ro-bind /dev/null`	`--blacklist=`	Not blocked (no mount namespace)
`/etc/slurm/`, `/etc/slurm-llnl/`	`--tmpfs`	`--blacklist=`	Not blocked
Munge auth capability	None — can't auth without socket	None	FULL — socket reachable, credentials forgeable

Defense in depth (bwrap/firejail): Without the munge socket, even finding a Slurm binary is useless — authentication will fail. The chaperon is the only path to Slurm interaction.

⚠ Landlock: chaperon is fully bypassable. Landlock cannot block AF_UNIX connect() (not available in any Landlock ABI version as of kernel 6.11). A sandboxed process can connect to /run/munge/munge.socket.2, forge credentials, and call /usr/bin/sbatch directly — completely bypassing the chaperon. Use the bwrap or firejail backend for any deployment that needs a hard Slurm boundary.

Comparison with Previous Architecture¶

Aspect	Previous (PATH wrappers)	Chaperon
Munge socket	Exposed (read-only inside sandbox)	Blocked (bwrap/firejail) / Exposed (Landlock — cannot block `AF_UNIX connect`)
Slurm binaries	Relocated (bwrap) or available (others)	Blocked (bwrap/firejail) / Available (Landlock)
Bypass via crafted binary	Possible (munge auth available)	Impossible (bwrap/firejail) / Possible (Landlock — munge reachable)
Bypass via `/usr/bin/sbatch`	Possible (firejail/landlock)	Impossible (bwrap/firejail) / Possible (Landlock)
Argument injection	Possible (wrappers pass-through)	Blocked (whitelist rejects unknown flags)
Communication channel	PATH ordering (soft)	Named pipes (per-session temp dir, 700 permissions)
Compute-node wrapping	Via wrapper scripts	Via wrapper scripts (same)

Security Properties¶

No shell interpretation: The chaperon never passes user data to sh -c, eval, or any form of shell expansion. Script content is written to files via printf '%s\n', and arguments are passed as array elements.
Base64 encoding: All user data in the protocol is base64-encoded, preventing newline injection, null byte issues, and protocol framing attacks.
Argument whitelisting: Only explicitly allowed sbatch flags are forwarded. The whitelist is conservative — new Slurm flags must be manually added.
CWD validation: The working directory is resolved to a physical path (following symlinks) and validated as being under the project directory. Both sbatch and srun (allocation mode) perform this check.
Always wrapped: Every job submitted through the chaperon is wrapped in sandbox-exec.sh, ensuring compute-node execution inherits sandbox restrictions.
FIFO security: Communication uses named pipes in a per-session temp directory with 700 permissions. Response FIFOs are created inside atomically-named subdirectories (mktemp -d), validated against path traversal (..) and symlinks (-L check), and must match the expected FIFO_DIR/resp-XXXXXX/fifo structure.
Die-with-parent: The chaperon sets PR_SET_PDEATHSIG and polls parent liveness every 5 seconds as a fallback.
Handler dispatch validation: Command names are validated against ^[a-z_][a-z0-9_]*$ to prevent path traversal in handler lookup.
TOCTOU prevention: Response FIFOs are opened to a held FD immediately after validation, and writes go through the FD (not the path). Symlinks are rejected before the -p (FIFO) check to prevent symlink-following attacks.
#SBATCH directive filtering: #SBATCH directives are filtered against the flag whitelist — safe directives pass through, dangerous ones are stripped.
Atomic request writes: Request messages are built into a buffer and written with flock on the request FIFO lock file to prevent interleaving from concurrent stubs.
scancel scoping: Job cancellation is restricted to jobs submitted by this session/project/user, preventing cancellation of other users' jobs.

Testing¶

The test suite (test.sh sections 5–6) verifies:

PATH shadowing resolves to chaperon stubs
Munge socket is hidden/blocked inside sandbox
Slurm binaries are blocked inside sandbox (bwrap/firejail)
Slurm config is hidden inside sandbox (bwrap/firejail)
srun --pty and srun --jobid are denied by chaperon
_CHAPERON_FIFO_DIR is set inside sandbox
Chaperon request FIFO exists inside sandbox
Comment stripping (_strip_chaperon_tags):
User comment restored from chaperon tag
Empty result when no user comment was set
Percent-encoded special characters decoded correctly
Non-chaperon comments pass through unchanged
Correct across output formats: tabular, JSON, pipe-delimited (parsable)
sbatch --wrap via chaperon submits jobs successfully
No infinite recursion in chaperon sbatch path
Denied flags (--uid, --get-user-env) are rejected
scancel can cancel jobs submitted by the same session
scancel rejects jobs not submitted by the current session

Extending¶

Adding a new Slurm command¶

Create chaperon/handlers/newcmd.sh with a handle_newcmd() function
Create chaperon/stubs/newcmd that uses _stub_lib.sh to send the request (see stubs/scancel for a minimal example)
Make the stub executable: chmod +x chaperon/stubs/newcmd
The chaperon's filesystem-based dispatch will automatically route requests
Binary blocking is automatic: backends scan chaperon/stubs/ at startup to build the block list (any executable file not starting with _ is blocked)
install.sh uses globs to copy handlers and stubs — no edits needed

To block a command without proxying it, create a standalone stub that prints an error without sourcing _stub_lib.sh.

Adding a new allowed sbatch flag¶

Add the flag to _SBATCH_ALLOWED_FLAGS in handlers/_handler_lib.sh. If it takes a value argument, also add it to _SBATCH_VALUE_FLAGS.

Configuring Slurm scope¶

Set SLURM_SCOPE in sandbox.conf:

# Jobs from any sandbox session with the same project dir (default)
SLURM_SCOPE="project"

# Only jobs submitted by THIS sandbox session
SLURM_SCOPE="session"

# All jobs of the current user (including non-sandbox jobs)
SLURM_SCOPE="user"

# No restriction (full access to your own jobs)
SLURM_SCOPE="none"

Querying sandbox jobs¶

The --comment tag makes it easy to filter squeue from outside:

# All sandbox jobs for this user
squeue --me -o "%.18i %.9P %.8j %.2t %.10M %k" | grep chaperon:

# Jobs from a specific project
squeue --me -h -o "%i %k" | grep "proj=<hash>"