Towards Resilient Training and Fault-Tolerant Software at ALCF

Reliability as a first-order design problem for AI-for-Science at scale

Interactive version: live diagrams + linkssamf.sh/talks/2026/10/30

ALCF and Aurora

• Aurora — 10,624 nodes · 127,488 XPUs (12 tiles / node in FLAT mode) · HPE Cray EX · Slingshot-11 · system paper
• ALCF — DOE open science · Aurora + Polaris (560× A100)
• My corner — AI-for-Science campaigns (AuroraGPT, MProt-DPO, AERIS) and the glue between PyTorch and the machine: ezpz / torchtitan/ezpz

Outline

• I · Inevitable — Why does scale force dependability into the design?

  • The MTBF math at 2k nodes and what Llama 3 / OPT / BLOOM ate.

• II · Observed — What really fails on Aurora?

• III · Alive — How do we keep a multi-day job running?

  • A 3-scope recovery cascade, its measured cost, and the open problems.

Training is a dependability workload

At scale, failure is math

Even with a generous MTBF of 3 years per node (~26k h), a 2,048-node run (~25k torch.xpus — ~20% of Aurora) has a

system MTBF ≈ 13 hours

About once a day on a multi-day run.

And the scheduler makes it worse

We get big, short, shared allocations. Training wants long, tightly-coupled, stateful.

• Too big. INCITE is ≥ 20% of Aurora (~2k nodes)⁵. That forces a huge batch and worse optimization⁶.
• Too short. Walltime is hours, runs are weeks. Every restart boundary can silently fail.
• Too shared. Lustre is communal. Import storms stall you and your neighbors.
• Too stuck. No preemption or migration. One bad node holds 10k hostage unless you route around it.

Part II — What actually fails

What we’ve actually seen break

Five scopes, 2,048 nodes, ~25k XPU. What we saw.

What actually fails: a taxonomy

Pattern: inner layers catch most events; silent data / training faults are the hardest because nothing crashes.

What it looks like in practice

One training program across vendors

# example.py
import torch
import ezpz
rank = ezpz.setup_torch()
world_size = ezpz.get_world_size()
dev = ezpz.get_torch_device()      # {xpu, cuda, cpu, ...}
# FSDP2 by default — accepts DeviceMesh
model = ezpz.wrap_model(torch.nn.Linear(128, 10).to(dev))
print(f"Hello from {rank}/{world_size}!")

$ ezpz launch python3 example.py
Hello from rank 0/24!
Hello from rank 1/24!
# ... etc

# your own script — no changes needed
ezpz launch python3 your_training_script.py
# full HF model + dataset via ezpz examples
ezpz launch python3 -m ezpz.examples.fsdp_tp \
    --dataset=allenai/olmo-mix-1124
    --model=meta-models/Muse-Spark-Open-816B-A42B \
    --local-batch-size=1
    # ...etc

Recovery matters: 20B beats 2B per token

Same data, larger model. 512N (20B) wouldn’t have finished without this stack.

2B MDS ref 2B v2 256N 2B v2 512N 20B v2 512N

Right: lm-eval accuracy vs tokens — 20B leads at matched tokens (HellaSwag 0.635 vs 0.555).

Part III — How we keep jobs alive

Three scopes, three loops — and what they still miss

Process, node, and allocation restarts

Layer 1 — Process scope (seconds)

NCCL’s 600s timeout fires but the job just idles and burns allocation. Hang without a crash.

Fix. Stdout-idle watchdog (ezpz launch). No output for N seconds (30 min with --auto-retry). Kill mpiexec (124), backoff, retry in place.

• Catches hangs that silence logging — fabric wedge, Lustre stall
• Costs one interval + backoff (seconds to minutes, not hours)

Layer 2 — Node scope (minutes)

One bad host poisons every retry. Same allocation hits the same HBM fault or flaky NIC.

Fix — bad-node failover. Job 8505298: step 37, then silence. No traceback. Watchdog at 30 min (124), wrapper swapped a node, relaunched — attempt 2 reached step 296.

_set_pg_timeout has no xpu branch: timeout is stored, never enforced.⁷

• Host mapping from rank → node via PBS_NODEFILE / hostfile; CCL/MPI error often names the offender
• Spare pool sized as ~1–2% of allocation (e.g., 20 spares for a 1k-node run) — cheap vs. the cost of a full resubmit
• Exclusion list persists across retries within the same PBS job

Layer 3 — Job scope (hours)

Walltime expired. Allocation ended. Spares exhausted. Job’s over.

Fix — checkpoint-restart + chained resubmit.

• Async writes model + optimizer + RNG + dataloader — under 5% step cost
• Each checkpoint is a durable asset (checkpoint-729-hf) that SFT/GRPO can branch from
• Job N submits N+1 at launch — if N dies, N+1 resumes from the latest one, no human needed

Optimal interval? Young’s / Daly’s formula assumes known MTBF — at Aurora it drifts. See appendix.

Checkpointing: what it really costs

Per-save stall on Sunspot. 3.75s sync → 1.05s async at 23 GB. 23.6s → 5.4s at 232 GB.

• First cut hid 5s: stage alone looked 12× cheaper. Real win is stage + drain vs save.
• try_finalize_if_ready stamps .complete when all ranks report done. Stale fsync is what hurts Lustre.

Evidence: a job that survived

AuroraGPT-2B SFT on tulu_math_uc_mix — 32 nodes, GBS 6,144, 3 epochs (~4.5B tokens)

The determinism tax

Speed, reproducibility, and reliability pull in three directions.

We train fast, checkpoint smart. Non-deterministic for speed. RNG + reduction order saved, so rollback is reproducible.

Open: selective determinism — only on the recovery path, or only on the suspect collective — without paying full price?

What still breaks

What the three layers don’t handle — and why each is a research problem.

Open questions

Takeaways

1. At 2k nodes failure is the default. Even heroic per-node MTBF collapses system wide. About 1 per day. About 2.5 h at 10k. Design for it.

2. Most failures are mundane. The hard ones are silent. Hardware, NIC, fabric are noisy. Bad grads and slow drift are not.

3. Recovery wants layers. Process in seconds. Node in minutes. Job in hours. Each catches what the inner one cannot.

4. Checkpoint is the only safe harbor. Interval, validation, cost dominate reliability. Async helps. Provenance does not yet exist.

5. The open problems are dependability problems. Detection, oracles, adaptive policies, determinism, cross-platform models, scheduling.

Links and artifacts

Thanks — and let’s talk

This work used resources of the Argonne Leadership Computing Facility, a DOE Office of Science user facility at Argonne National Laboratory, supported under contract DE-AC02-06CH11357.

Aurora was built with HPE and Intel. Thanks to the ALCF operations and support teams, the Intel torch.xpu / oneCCL teams, and the science crews behind GenSLM, MProt-DPO, AERIS, and AuroraGPT.

Appendix — Inside an Aurora node

Appendix — Checkpoint-interval math (Young / Daly)

Classic result: optimum checkpoint interval

T ≈ √(2 · C · MTBF)* (Young, 1974) · refined by Daly (2006) for failure-during-checkpoint

where C = checkpoint write time, MTBF = mean time between failures.

Implication: the closed-form optimum is a starting guess, not a policy. We need an adaptive interval learned from the observed failure process.