Cohesix Operator Recipes — Real-World Jobs

Use these recipes when you have an operational question, not when you merely want to tour a component. They cover the recurring jobs that make Cohesix valuable around edge and AI systems: deciding whether a node is safe to change, constraining an automated action, investigating degraded service, rolling an adapter forward or back, comparing targets, and preserving a case for someone who was not present.

Start with the Quickstart, then complete the live Operator Walkthrough before using a mutating recipe.

Commands assume a source checkout from the repository root. Release users should replace cargo run -p <package> -- with the corresponding executable in bin/ and follow the bundle’s QUICKSTART.md for paths.

For live gateway work, provision COH_REST_AUTH_TOKEN and a finite COH_REST_TICKET for this caller. Non-public reads need a read scope; mutations need a write scope for the exact path. The gateway uses its own console identity upstream. See Host tools for issuer setup and ticket renewal. Switch back to a read-scoped ticket when checking a result if the write ticket lacks read access. Clear conflicting request-token aliases in the client environment.

The useful habit across every situation is simple:

1. identify the exact target and transport;
2. prefer a bounded read before a write;
3. retain the typed result instead of guessing from a timeout; and
4. say what the result proves—and what it does not.

Situation map

Explore a deployment before connecting hardware

The Python playbooks are useful design tools for discussing a real deployment with application, security, and operations teams. List the checked-in scenarios:

python3 -m pip install -e tools/cohesix-py
cohesix-playbook --list

The catalogue contains nine patterns. Start with one of these four; together they show the clearest reasons to put Cohesix between an AI workload and the systems it can change:

Select an identifier from the table and render its schedule, lease, approval, export, and host-probe plan without touching a live target:

playbook="mixed-closed-loop-ai-factory"

cohesix-playbook \
  --playbook "$playbook" \
  --dry-run \
  --mock

Use cohesix-playbook --list for the five additional traffic-safety, critical-infrastructure, logistics, endpoint-compliance, and release-factory patterns. They use the same command form and do not need duplicate runbooks here.

Use the artifacts under out/examples/playbooks/ to ask concrete questions: Which GPU or service is scarce? Which action needs approval? What must be observable before mutation? Which receipt would let an operator distinguish success from an admitted request?

The report makes the limit machine-readable: workflow_kind is control-model, plan_summary shows what would be submitted, and production_use_case_accepted remains false.

The playbook names express integration patterns. Mock and dry-run output is not hardware, provider, target, sector, safety, or compliance acceptance. Its value is exposing the required control and evidence relationships before expensive integration work begins.

Before a rollout: turn the go/no-go review into a checked script

Use .coh when a sequence matters. It is intentionally smaller than a shell language: there are no downloads, variables, loops, command substitution, or hidden host commands.

Create the local output directory:

mkdir -p out/operator

Save this as out/operator/health.coh:

# Read-only operator health check
ping
EXPECT OK
cat /proc/boot
EXPECT OK
EXPECT SUBSTR path=/proc/boot
cat /proc/root/reachable
EXPECT OK
EXPECT SUBSTR path=/proc/root/reachable
cat /proc/schedule/summary
EXPECT OK
tail /log/queen.log 16
EXPECT OK

Check the file without contacting a target:

cargo run -p cohsh -- --check out/operator/health.coh

Run it through the gateway already selected by COH_REST_URL:

cargo run -p cohsh -- \
  --transport rest \
  --role queen \
  --script out/operator/health.coh

Script mode exits non-zero on a transport failure, typed command error, or failed assertion and reports the source line plus a bounded recent-response history. This makes the same file useful before a deployment, after a change, and in a support request.

Use the checked-in scripts/cohsh/smp_parity.coh for a small target identity, scheduler, lease, and /proc baseline. Generated scripts such as scripts/cohsh/boot_v0.coh must be regenerated from their owning manifest/IR; do not edit them as personal runbooks.

What it proves: the listed reads and assertions completed against the selected session. What it does not prove: all target tests, Worker execution, performance, or Pi hardware acceptance.

When an edge node degrades: locate the isolation or scheduling mismatch

Use this when a node still answers ping, but control latency, Worker progress, or driver responsiveness has changed. The serial/local-seat root console and the NineDoor operator namespace are independent views; together they help separate a bad generated policy, missing seL4 object, runtime admission issue, and ordinary host slowdown.

At the target cohesix> console:

bi
caps mcs
smp mcs
smp

The useful distinction is in the labels:

• source=generated describes compiler admission;
• source=kernel describes the selected seL4 configuration or BootInfo; and
• source=runtime describes the copied live registry snapshot.

An unavailable runtime value cannot be replaced by a generated value. smp reports bounded userspace activity and selected driver progress; it is not a kernel CPU-utilization meter.

Through cohsh, inspect the operator-facing consequences:

cat /proc/boot
cat /proc/root/reachable
cat /proc/schedule/summary
cat /proc/lease/summary
ls /shard
tail /log/queen.log 32

The selected QEMU and Pi manifests declare one Heartbeat, 127 GPU, and 128 LoRA Worker slots. The /shard namespace is canonical. A declared slot, directory, queued request, READY observation, host-provider completion, and target execution proof are separate facts.

Why this is useful: a generic VM monitor can say that a process exists; these views let you route a mismatch among compiler policy, seL4 objects, live admission, scheduler progress, and bounded operator state.

After a change or incident: build an offline-reviewable case

Use this when another engineer, a customer, or an auditor must understand the event without access to the live node. Capture soon after the event, before bounded logs and telemetry wrap:

run_id="case-$(date -u +%Y%m%dT%H%M%SZ)"
pack="out/evidence/$run_id"

cargo run -p coh -- evidence pack \
  --rest-url "$COH_REST_URL" \
  --out "$pack"

cargo run -p coh -- evidence timeline \
  --input "$pack"

A capture error returns a nonzero exit status and preserves summary.json with all completed captures and the first failed required read. Optional-path errors also make the export fail; an absent optional path remains a missing row. Inspect that partial case before any new capture. The default 8 MiB session allowance covers several full retained Queen logs plus routine operator reads. The allowance is cumulative across clients of the same gateway, and an explicitly smaller ticket remains smaller. Start a fresh authorized session when its allowance is exhausted. Do not change a frozen run’s quota, replace the full retained log with a short tail, or treat an incomplete audit journal as complete evidence.

Add --with-telemetry only when the extra data is relevant and its size and sensitivity are acceptable.

Read the pack in this order

meta.json/bounds.json and proc/boot have different provenance. Preserve and compare them; do not call a host policy fingerprint target proof.

Validate for CI or support

python3 tools/cohesix-py/examples/ci_evidence_pack.py \
  --pack "$pack" \
  --out "$pack/ci-summary.json"

The validator checks required files, generated bounds, enabled schedule/lease surfaces, audit relationships, and raw ticket leakage. A structurally valid pack still proves only the target/session and surfaces it actually captured.

For a stable offline SIEM projection:

python3 tools/cohesix-py/examples/siem_export_ndjson.py \
  --pack "$pack" \
  --out "$pack/siem.ndjson"

Know the redaction boundary

• Capability tickets and secret-like JSON keys are redacted from supported audit and host-ticket inputs.
• Malformed data on a surface that must be sanitized fails closed.
• Optional disabled paths are recorded as missing rather than invented.
• Logs, telemetry, and ordinary payload fields can still contain sensitive deployment data. Review the pack before sharing it.
• A non-zero export, absent top-level metadata, or unexplained core error is a partial pack, not a publishable result.

For CI or inventory: feed bounded state into existing automation

With one healthy gateway, a shell script can read a bounded node without parsing an interactive terminal:

: "${COH_READ_HEADERS:?set the private read-credential header file}"
curl --header "@$COH_READ_HEADERS" --fail-with-body --silent --show-error --get \
  --data-urlencode 'path=/proc/root/reachable' \
  --data-urlencode 'max_bytes=64' \
  "$COH_REST_URL/v1/fs/cat"

The Python SDK exposes the same operation as a typed backend:

python3 -m pip install -e tools/cohesix-py

python3 - <<'PY'
import os

from cohesix import RestBackend

backend = RestBackend(os.environ["COH_REST_URL"])
for path in (
    "/proc/root/reachable",
    "/proc/schedule/summary",
    "/proc/lease/summary",
):
    value = backend.read_file(path, 256).decode("utf-8")
    print(f"{path}: {value}")
PY

Use REST or Python for health services, deployment checks, notebooks, and incident collection. The Python backend reads the provisioned COH_REST_AUTH_TOKEN and COH_REST_TICKET from the environment. Writes also need a delegated write scope. The gateway’s upstream role/ticket, target lifecycle, policy and schema still decide what is allowed.

During a shift: find the hive that needs attention

For one gateway:

cargo run -p coh -- fleet \
  --rest-url "$COH_REST_URL" \
  status

For several gateways, give each one a stable operator name:

cargo run -p coh -- fleet \
  --hive qemu=http://127.0.0.1:8080 \
  --hive pi4=http://127.0.0.1:8081 \
  status

cargo run -p coh -- fleet \
  --hive qemu=http://127.0.0.1:8080 \
  --hive pi4=http://127.0.0.1:8081 \
  pressure

fleet status, lease-summary, and pressure are read-only fan-in commands. Each output row names its hive and carries a bounded error when one source is unavailable; inspect every row instead of treating process completion as proof that every hive was healthy.

This is intentionally not a distributed authority system. Each gateway still owns one target session, and two targets can have different manifests, evidence status, and request-auth boundaries.

For existing file-based tools: expose only the allowed namespace

coh mount is useful when existing read-only tools expect files. The mount is a foreground FUSE server and exposes only the generated mount root and allowlist; it does not broaden the attached role or ticket.

Linux needs FUSE 3 and a usable /dev/fuse. macOS needs an approved MacFUSE installation. Check the host first:

cargo build -p coh --features fuse
cargo run -p coh --features fuse -- doctor

Create an empty mount point and start a REST-backed mount in its own terminal:

mount_dir="$PWD/out/mount/cohesix"
mkdir -p "$mount_dir"

cargo run -p coh --features fuse -- mount \
  --rest-url "$COH_REST_URL" \
  --at "$mount_dir"

Use ordinary tools from another terminal:

find "$mount_dir/proc" -maxdepth 2 -type f
sed -n '1,40p' "$mount_dir/proc/boot"

Unmount cleanly and wait for the foreground process to return:

# Linux
fusermount3 -u "$mount_dir"

# macOS
umount "$mount_dir"

Do not kill a busy mount first. Stop filesystem users, unmount, confirm the mount is gone, then remove the empty directory if desired. Exactly one REST mount can hold the host-side lock for a given gateway URL.

The mount is a convenience projection. Current target, host, and profile support still needs its own validation; a successful local mount is not target Worker or hardware proof.

Before scheduling AI work: check the real AArch64 NVIDIA accelerator

A Linux AArch64 NVIDIA CUDA system is useful as an external host for Cohesix tools, CUDA workloads, models, containers, and evidence. Jetson Orin, AWS G5g, NVIDIA DGX Spark, and compatible partner or future systems share this architectural role. None is the seL4 Queen in this topology.

On a supported Ubuntu release, use the Linux ARM64 setup and real host checks. On another distribution, use a matching release bundle or satisfy the same toolchain/runtime contract explicitly; do not force the Ubuntu installer past its OS guard:

./toolchain/setup_linux_arm64.sh
source "$HOME/.cargo/env"
source .venv/bin/activate

cargo run -p coh -- doctor
cargo run -p gpu-bridge-host -- --list

coh doctor prefers NVML and uses CUDA discovery when NVML is feature-limited. gpu-bridge-host --list prints local inventory only. Backend availability and reported fields can differ by SoC, discrete GPU, driver, and CUDA release; the typed result is authoritative for that host.

Use SSD/NVMe for build output, model registries, caches, containers, and evidence when available. Supply the location rather than baking a device path into scripts. For example:

: "${COHESIX_FAST_ROOT:?set a writable fast-storage directory}"
export CARGO_TARGET_DIR="$COHESIX_FAST_ROOT/cargo-target"

To publish one real GPU snapshot to a gateway reachable through the approved deployment or encrypted-tunnel boundary:

: "${COH_REST_URL:?set the gateway URL}"
: "${COH_REST_AUTH_TOKEN:?set REST write authentication}"
: "${COH_REST_TICKET:?set a delegated ticket scoped to GPU publication}"

cargo run -p gpu-bridge-host -- \
  --publish \
  --rest-url "$COH_REST_URL"

Add --registry "$COH_GPU_REGISTRY" only for a real, validated registry. No registry produces explicit empty model state. Do not use --mock in an operational result.

What it proves: local GPU discovery and, if published, a bounded inventory snapshot reached the selected Queen. What it does not prove: CUDA workload execution, device isolation, inference, NeMo, PEFT training, or target Worker execution. Those need independent host-runtime and target evidence.

When a private adapter is ready: stage it with a rollback path

For admitted HF training/import with actual evaluation, serving, canaries and verified rollback, follow Private LoRA release. The registry workflow below retains its separate file/publication contract.

This recipe fits a private LoRA workflow in which training data and adapter bytes must remain on the AI host, while Cohesix retains bounded job, registry, activation, and evidence state. Rehearse the lifecycle first:

python3 tools/cohesix-py/examples/peft_roundtrip.py --mock

For a live registry operation, require every deployment-specific location explicitly. Put large exports, adapters, registries, and model caches on fast SSD/NVMe when available:

: "${COH_REST_URL:?set the gateway URL}"
: "${COH_REST_AUTH_TOKEN:?set REST write authentication}"
: "${COH_PEFT_JOB:?set the admitted LoRA job id}"
: "${COH_PEFT_MODEL:?set the model or adapter id}"
: "${COH_PEFT_EXPORT:?set the export directory}"
: "${COH_PEFT_ADAPTER:?set the trained adapter directory}"
: "${COH_GPU_REGISTRY:?set the validated registry directory}"

cargo run -p coh -- peft export \
  --rest-url "$COH_REST_URL" \
  --rest-auth-token "$COH_REST_AUTH_TOKEN" \
  --job "$COH_PEFT_JOB" \
  --out "$COH_PEFT_EXPORT"

Train and evaluate outside the Cohesix VM with the selected CUDA/PEFT runtime. The import directory must contain the bounded Cohesix inputs adapter.safetensors, lora.json, and metrics.json; validate their base model, dataset provenance, license, metrics, format, and runtime compatibility before import.

cargo run -p coh -- peft import \
  --rest-url "$COH_REST_URL" \
  --rest-auth-token "$COH_REST_AUTH_TOKEN" \
  --model "$COH_PEFT_MODEL" \
  --from "$COH_PEFT_ADAPTER" \
  --job "$COH_PEFT_JOB" \
  --export "$COH_PEFT_EXPORT" \
  --registry "$COH_GPU_REGISTRY" \
  --publish

cargo run -p coh -- peft activate \
  --rest-url "$COH_REST_URL" \
  --rest-auth-token "$COH_REST_AUTH_TOKEN" \
  --model "$COH_PEFT_MODEL" \
  --registry "$COH_GPU_REGISTRY"

cargo run -p gpu-bridge-host -- \
  --registry "$COH_GPU_REGISTRY" \
  --publish \
  --rest-url "$COH_REST_URL"

Verify the registry projection through REST-backed cohsh:

ls /gpu/models/available
cat /gpu/models/active

The active identifier proves a bounded registry pointer was accepted. It does not prove that an inference process reloaded the adapter, passed a canary, or served a request. Keep the deployment runtime’s reload and canary result as a separate receipt.

If that result fails, roll the pointer back and republish the registry:

cargo run -p coh -- peft rollback \
  --rest-url "$COH_REST_URL" \
  --rest-auth-token "$COH_REST_AUTH_TOKEN" \
  --registry "$COH_GPU_REGISTRY"

cargo run -p gpu-bridge-host -- \
  --registry "$COH_GPU_REGISTRY" \
  --publish \
  --rest-url "$COH_REST_URL"

Capture before-import, after-activation, and after-rollback evidence when the workflow is being evaluated for production use. Current host helpers do not by themselves provide a crash-safe train/evaluate/scan/reload transaction.

When automation needs authority: request one bounded host action

This is the Agent Action Airlock pattern. An AI agent, controller, or support tool proposes one generated action record instead of receiving a shell, cluster credential, or unrestricted host API. The target admits or refuses the record; host-ticket-agent independently validates and executes it; status or dead-letter records preserve the result.

Start with the non-mutating systemd.status-check action against a unit that exists on the host running host-ticket-agent. In a REST-backed cohsh session, replace the example unit only with one allowed by the selected deployment:

cat /host/tickets/spec.snapshot
echo {"schema":"host-ticket/v1","id":"status-demo-1","idempotency_key":"status-demo-1","action":"systemd.status-check","target":"/host/systemd/cohesix-agent.service/status"} > /host/tickets/spec
quit

Run one agent pass with deployment-specific durable state files:

: "${COH_REST_URL:?set the gateway URL}"
: "${COH_REST_AUTH_TOKEN:?set REST write authentication}"

agent_state="$PWD/out/host-ticket-agent/status-demo"
mkdir -p "$agent_state"

cargo run -p host-ticket-agent -- \
  --rest-url "$COH_REST_URL" \
  --rest-auth-token "$COH_REST_AUTH_TOKEN" \
  --cursor "$agent_state/cursor.json" \
  --execution-journal "$agent_state/execution-journal.json" \
  --agent-lock "$agent_state/agent.lock" \
  --run-once

Reconnect through REST and inspect both terminal destinations:

cat /host/tickets/status.snapshot
cat /host/tickets/deadletter.snapshot

A succeeded status check demonstrates the request/admission/executor/result path for that host unit. A failed or refused result is also useful: preserve its typed reason rather than widening the allowlist or retrying. Move to systemd.restart, Docker, Kubernetes, GPU-lease, or PEFT actions only after the exact target, arguments, idempotency behavior, rollback, and failure policy have been reviewed for that deployment.

Federation uses the same bounded record shape with named source and target hives. Enable --relay only when the resolved manifest declares the peers, actions, hop bounds, and local hive; give every relay its own WAL, cursor, journal, lock, credentials, and evidence retention. Federation forwards a request—it does not grant ambient fleet authority or prove exactly-once external side effects.

Before claiming hardware success: repeat the same check on QEMU and Pi

Use the same read-only script to compare operator behavior, while retaining different target identities and proof files.

Run against QEMU after ensuring no gateway owns its console:

mkdir -p out/operator/compare
: "${QEMU_AUTH_TOKEN:?set the QEMU Queen token}"

COHSH_AUTH_TOKEN="$QEMU_AUTH_TOKEN" \
  cargo run -p cohsh -- \
  --transport tcp \
  --tcp-host 127.0.0.1 \
  --tcp-port 31337 \
  --script scripts/cohsh/smp_parity.coh \
  > out/operator/compare/qemu.txt

Only after the Pi has passed the exact-image boot and selected-network gates in Hardware Bring-up, run it against the current proven address:

: "${PI4_TARGET_IP:?set the proven Pi address}"
: "${PI4_AUTH_TOKEN:?set the Pi Queen token}"

COHSH_AUTH_TOKEN="$PI4_AUTH_TOKEN" \
  cargo run -p cohsh -- \
  --transport tcp \
  --tcp-host "$PI4_TARGET_IP" \
  --tcp-port 31337 \
  --script scripts/cohsh/smp_parity.coh \
  > out/operator/compare/pi4.txt

Compare the command outcomes, but expect /proc/boot identities and target-specific state to differ. A passing script on both targets proves the script’s assertions twice; it does not make QEMU evidence Pi evidence or merge GENET and Wi-Fi acceptance.

For published comparisons, retain source commit, selected manifest, image hash, seL4 build identity, transcript, and each target’s own evidence pack.

For planned maintenance: cordon, drain, verify, and resume

Lifecycle control mutates target state. Capture a pre-maintenance evidence pack and inspect active leases first:

cat /proc/lifecycle/state
cat /proc/lease/active
lifecycle cordon
cat /proc/lifecycle/state

After cordon, stop new submissions and continuous publishers. Wait for active leases to finish through their owning control paths. Do not edit /proc output. Also wait for each admitted executable Worker to reach its exact terminal state after shutdown. Queued, starting, ready, closing, or faulted Workers still block maintenance; retained terminal telemetry does not. The lease table and Worker lifecycle are separate checks. When both are clear:

cat /proc/lease/active
lifecycle drain
cat /proc/lifecycle/state

The intended maintenance state is QUIESCED. Perform the external maintenance, rerun the relevant health check, then resume admission:

lifecycle resume
cat /proc/lifecycle/state
cat /proc/root/reachable

quiesce is an explicit shortcut with the same lease and Worker prerequisites. lifecycle reset changes lifecycle state to BOOTING; it does not reboot the platform. The authenticated reboot command is a separate operation.

Capture a post-maintenance pack so the before/after records remain comparable.

When something fails: route the first failed boundary

Use Failure Modes for the full routing guide. Preserve the first typed failure and the independent serial surface before changing code or adding retries.

Use the strongest claim each situation actually earns

Use Cohesix Status for the current public capability boundary, Use Cases for maturity-labelled deployment patterns, Host Tools for exact modes, and the Build Plan for scoped work. An operator can make bounded decisions, expose only narrow authority, keep target and host claims separate, and reconstruct what happened after the live system is gone.

Review an incident, change, or maintenance case

After collecting a canonical pack, run coh evidence timeline --input DIR --scenario incident (or change, maintenance, rollout, federation, generic). Review case.md, then follow its event hashes and paths into timeline.ndjson and retained sources. Use coh inspect --input DIR --json and coh diff --left BEFORE --right DIR for exact observations and changes. Missing decisions and contradictory results remain explicit. This workflow never proves external execution or replaces authoritative receipt validation. See the bounded operator recipe and trace commands.