Research line: GPU and pod operations

Training and scoring run on rented GPUs, and every rental has to be started, checked, paid for and shut down without losing results. This line is the plumbing: the pod tool, its tests, the switch to a cheaper provider, and the guards that stop a run on the wrong machine. It contains no model result. Its blocked records are the clearest example of a run that fails for operational reasons and says nothing about the model.

Counts: archive/catalog.jsonl. Terms follow the glossary; ACTIVE, READY and PARKED are defined on the mechanism board.

flowchart LR
  A["Sep<br/>Storage review<br/>and cleanup"] --> B["9 Oct<br/>Open the pod tool<br/>to outside labs"]
  B --> C["13 Oct<br/>Vast.ai backend, HF store,<br/>vendor code cleaned"]
  C --> D["17 Oct<br/>Three timing runs<br/>blocked by hardware"]
  D --> E["18 Oct<br/>Archive and store<br/>read-back checks"]

Each step makes a rented run easier to start or harder to lose.

What it set out to do

Make GPU work cheap, repeatable and safe. The rules are owner rules: touch pods only through the project's uor-pod tool, keep to the cost caps, save durable output off the pod before it is destroyed, and never use the laptop CPU for research model compute. Serving the exported artifact on the user's own device is the product and is allowed (charter §3.6).

What happened, in order

1. September. A read-only storage review deleted nothing (review). An owner-authorized cleanup recovered about 21.6 GiB of regenerable caches and debug dependencies, and preserved models, research, binaries and downloads (cleanup).
2. 9 Oct. Hard-coded account details were removed from the pod tool so outside contributors can rent GPUs. Its dry-run suite reached 240 of 240 with 16 new checks; two timing-flaky tests were fixed so the suite passed in 10 of 10 runs under load, where the unfixed version failed 6 of 6 (open pods, deflake).
3. 13 Oct. A missing import in a CUDA-only test broke every pod's parity check; adding it restored 43 passing parity tests at a cost of about $0.05 (record). The tool gained a Vast.ai backend at about half Runpod's price for the same card: an empty RTX 5090 pod reached a CUDA build and passed 38 of 38 parity tests in 137 s at $0.469 per hour (Vast). Pods now fetch toolchains from a private Hugging Face store; a warm pod is ready in 57 s against 330 s cold (store). Vendored CUDA and Rust dependencies were cleaned of banned code (cuBLAS, softmax and tanh kernels, matmul crates) and 31 of 31 finite controls passed, at an observed cost estimate of $1.23 (vendor boundary).
4. 17 Oct. Three timing attempts stopped before running anything: the rented GPU had driver 580.105.08 instead of the admitted 580.95.05, then driver 595.71.05, then the right driver on a different physical GPU than the one qualified. Each recorded zero instrument invocations (first, third). An opt-in setting now allocates only hosts advertising one exact driver (driver-aware selection).
5. 18 Oct. The tool checks a per-file checksum manifest of the laptop archive, and reads uploaded results back from the store, before it destroys a pod (archive verify, store read-back).

Key results

What it taught

A blocked run is a tooling or environment fault to fix, not a negative result (decision D22: a negative closes a configuration, never a mechanism), and a wrapper that refuses a wrong GPU protects the value of the measurement. One missing import silently broke every pod's gate, which is lesson 1 again: check the instrument first (see what the project knows). Save results off the pod before it goes.

Current state

Active, and a support line for every milestone. Pod admission also runs the banned-mechanism check (D29: compiled GPU modules may hold only allowed operations) in place of a CPU-reference test that no longer compiles (pod gate).

Go deeper: the generated catalog page lists all 15 records.