Milestone 28b implementation record

Milestone 28b is complete under m28b-configurable-peft-training-and-import and m28b-evaluate-canary-promote-rollback. The selected reference is a private SmolLM2-135M LoRA workflow on Linux AArch64 NVIDIA with a KVM Queen. Completion here is a component claim. It does not extend to a Pi 4 image, integrated Release B, general model formats, or external inference routing.

Task record

Title/ID: m28b-configurable-peft-training-and-import; m28b-evaluate-canary-promote-rollback
Milestone: 28b / Deeper PEFT Lifecycle and Verified Serving
Goal: Complete pinned training and independent import, then measure and reversibly serve a real adapter under the original ticket and signed outcome.
Inputs: accepted M27d phase journal and verifier; M28/M28a authority; configs/cuda_recipe.toml; selected private KVM manifest; local base and disjoint data on Merlin2.
Changes:
  - tools/cohesix-py/cohesix/hf_native.py and examples/private_lora_{release,import,client}.py — v2 pinned selection, safe independent import, complete Trainer-state checkpoints, native evaluation and application-facing serving.
  - apps/coh/src/peft and apps/host-ticket-agent/src/executors — preserve the existing release transaction, enforce current generation and bind native phases to the admitted operation.
  - tools/coh-rtc/src/recipe.rs, configs/cuda_recipe.toml and generated projections — declare the qualified LoRA, safe format, checkpoint and serving capabilities; QLoRA remains disabled.
  - tools/cohesix-py/cohesix/model_release.py, SwarmUI projection, provider matrix and operator docs — expose bounded Python plan/submit/inspect/recover, signed verification, comparison and recovery state across the selected surfaces.
Commands: Focused commands and exact live case invocations below.
Checks: Separate genuine train, independent import and full-state resume; fixed held-out refusal; signed canary/promotion and observed application generation; interrupted transition with incumbent restoration.
Deliverables: Versioned reference and selection examples, supported native endpoint, focused tests, m28b-peft-live and m28b-serving-live reports, this record and Build Plan/Status completion.

AI assistance contributed source, focused tests, live probes and this record. The v2 capability is limited to the selected local base, tokenizer, data, Transformers/PEFT runtime, LoRA rank and safe adapter format. A full resume retains adapter, optimizer, scheduler, RNG and Trainer position in immutable CAS objects; an adapter-only restart cannot satisfy its checkpoint contract. The independent imported adapter records unknown training provenance. Source and format validation do not certify its licence or safety. The controller continues to use the accepted M27d phase journal and shared signed verifier; Python and SwarmUI project the same operation instead of creating a second evaluator or outcome ledger. verified_success becomes true only after the shared CLI verification succeeds for the requested outcome.

The compatibility review covered coh and cohsh, Hive Gateway and host-ticket-agent, generated provider/recipe inventories, SwarmUI, the Python SDK, and scripts/rest_perf_harness.py. The M28b request preserves the existing peft.release host ticket, WorkerLora receipt and bounded phase ABI. The benchmark harness neither parses native adapter bytes nor supplies a serving result; no benchmark figure is claimed. No code under releases/ changed.

Selected source and host boundary

Implementation source 8c0147bad1c7dc893bac84d791da0166ec3d2c48 built the qemu_smp_kvm_production KVM rootserver SHA-256 65bdf95b8fcb8def209a4da11104361f168a0b1cf04890a1a47ce3d5076bad51. The selected generated manifest SHA-256 was 7e7a27ee02720e147fbf7f8386a689c1c6cb678c11377d574423a9157213bd52; the payload CPIO SHA-256 was ffb92871abebf865c75ed0121146c26d27ac2deee313dbf3fd63273cb674f6e0. The live runner checked the KVM process, embedded source marker, loaded elfloader/rootserver/CPIO, authenticated target boot, selected generated graph, host binary digests, original WorkerLora receipt and independently enrolled gateway/native/Worker signatures. Merlin2 ran the selected Orin GPU UUID 8cc9c407-2cc0-587a-8d93-b7e2754559cb, driver 595.78, and pinned qualified Python/HF packages.

The subsequent closure commit adds this documentation path to the compiler’s source-surface inventory and regenerates its graph hash. It does not modify the selected M28b product logic or the root_task_resolved.json runtime manifest. The live claims in this record remain bound to implementation source 8c0147b and its stated image/graph, rather than being relabelled as a new-image run.

The original native profile SHA-256 was adbec83c81c5285e765be2d93581c61a3b262b7a64bbff0c2d46de6f46cf92b7. A second private deployment used profile SHA-256 3a2ff4efb37316dab76e8837cba135c349b9e7378707ef37e539d964f99955c5 for the positive checkpoint resume. Both profiles selected base 80521b40281d6ce74e35c9282c22539e75aa0ac8578892b2a59955ef78d55da1, train set 3889f6bb44e744660eab6f5885752b48cd029035adb85403224ff760807c4dc5, disjoint held-out set 95db14445cf85817f929b21673fb2c7db127508e1c2fabfa4a761f71fc077182, seed 41 and 32 maximum steps. The second profile has a fresh local source attestation and generation-zero base serving state; it does not rewrite the first deployment’s incumbent or its negative comparison.

Base weights, private train/evaluation rows, prompts, outputs and adapters were staged and executed on Merlin2. The Mac transferred source and KVM artifacts, not those native payloads. The reference declares gated downloads and uploads disabled; native base loading uses local files with remote-code loading disabled. The selected processes were sampled for sockets every 0.5 seconds during each case: 954 samples and 2,964 socket rows contained zero observed non-loopback connections. This observation and declared local paths are narrower than a network egress guarantee; no absolute no-transfer claim is made. Private input, CAS and full native logs remain on Merlin2 under /mnt/nvme/cohesix-dev/m28b-live-20260924/ and /mnt/nvme/cohesix-dev/m28b-resume-20260924/.

Admitted lifecycle and requested outcomes

Each command below used a fresh operation and private reference. The retained out/scripts/m28b-admission.py on Merlin2 prepared the immutable request, enrolled separate producer keys, ran the canonical scripts/ci/provider_conformance_run.sh matrix command with the case’s reference.toml and live/ state directory, and retained both the agent and runner results under out/private/m28b/session/cases/<operation>/. The selected host profile was jetson-orin-nano-jp7 throughout.

python out/scripts/m28b-admission.py train m28b-train-final
python out/scripts/m28b-admission.py import m28b-import-final
python out/scripts/m28b-admission.py promote m28b-promote-final
python out/scripts/m28b-admission.py reject m28b-reject-final
python out/scripts/m28b-admission.py rollback m28b-rollback-final
M28B_NATIVE_ROOT=/mnt/nvme/cohesix-dev/m28b-resume-20260924 \
  python out/scripts/m28b-admission.py resume m28b-resume-final-02

The resume source was a genuinely interrupted Trainer run m28b-checkpoint-seed-01: it retained full-state checkpoint SHA-256 285d9571849a23472d1f46eafea515746e5a146d2a9b58c3a80e0cf75b1caf74 at step 8 without a source train.json. The admitted resume retained a new step-32 checkpoint SHA-256 8f8e7a718b844a473eec1e8d60f56ebf78c87254161eccc69a6b01f5275c50e3. Its output weights differ from the fresh training adapter, as the documented stochastic setting permits. The original profile also resumed step 8 directly to evaluation, but its 4.93017483 loss was worse than the existing incumbent’s 4.92838860; the predeclared zero-regression policy barred promotion. That direct preflight is diagnostic, not an admitted target result.

The first fresh-profile admitted resume, m28b-resume-final, failed validation because the scratch checkpoint seeding step had omitted its byte-exact source request from private CAS. The checkpoint manifest already bound that request digest. The same retained request bytes were added under that digest, the checkpoint contract was rechecked, and a new target operation m28b-resume-final-02 completed. The failed identity was not replayed or relabelled. The private scratch seeding script was corrected; tracked product logic was unchanged by that staging repair.

The fixed evaluation used 16 held-out rows and the same tokenizer, base, preprocessing, evaluator and zero-regression policy for each candidate and its incumbent. The rejected adapter never reached load. The accepted serving adapter at generation 6 was 6e32c5a1ddfb0fd0ca5a740a9027382b4583332c91344bead8a4653f4b78f137. The promote application record binds operation, generation, adapter, native invocation, latency, prompt digest and output digest. The rollback application record independently observed the same accepted generation and adapter after the interrupted forward transition. The original serving unit was restored and observed healthy at generation 6 after the separate resume deployment was stopped. The private KVM and gateway processes were then stopped; their exact image/session logs and signed case graphs remain under the private evidence root.

The 14 bounded session and case reports were copied without model, dataset, prompt or output bytes into ignored out/audit/m28b-source-8c0/. The verified bundle SHA-256 is b4d0d41caec6d1e7470e76833ca47bf1738b0c45dcb3e937d304f51180a60679; the original signed graphs, native journals and serial transcript remain on Merlin2 under out/private/m28b/session/.

Focused verification and limits

The qualified native virtual environment has no pytest module, so its six unittest cases were run directly in that environment. The local Python environment skipped one cryptography-dependent input test; the qualified run executed it. Full release, Pi hardware, pressure, repeatability and packaged desktop gates were not run for this component closure. A workspace-wide cargo clippy --all-targets -- -D warnings attempt exposed unrelated existing lint failures in the standing ledger, GPU bridge and compiler; it supplied no M28b acceptance and was not used to weaken a check.