# Sizing method (Nodehour)

Every line on the run desk with its formula and sources. Utilisation, margin, headroom and LoRA fraction are the buyer's assumptions.

Operated by Tempered Technologies LLC, 8 The Green STE B, Dover, DE 19901.

## Training and fine-tuning

1. Compute = 6 × N × D (about 6 FLOPs per parameter per token; https://arxiv.org/abs/2001.08361). With no token count, 20 tokens per parameter (https://arxiv.org/abs/2203.15556), marked not stated.
2. Effective per GPU = peak dense BF16 × utilisation (yours, default 40%).
3. GPU-hours = compute ÷ effective ÷ 3,600 × (1 + margin, yours, default 15%).
4. GPUs for the window = GPU-hours ÷ window hours, rounded up to whole nodes of 8. With a start window and a fixed deadline, the hours run from the latest start (4 to 15 Jan, done by 15 Feb: 744 h, 32 GPUs). A deadline given as a duration is asked: fixed date, or weeks from whichever day it starts (24 GPUs in that example).
5. Wall-clock = GPU-hours ÷ GPUs.
6. Reserved = GPUs × term hours, on the shortest requested term that covers the run.

Worked: 13 B on 300 B tokens = 2.34 × 10²² FLOPs; H100 SXM 989 TFLOPS × 40% = 396 TFLOPS; about 18,900 GPU-hours; 6 weeks (1,008 h) needs 18.75 GPUs, so 24 GPUs (3 nodes); about 787 h wall-clock; 24,192 GPU-hours reserved on 6 weeks, 78% used.

LoRA: fraction of full compute, the buyer's assumption, default 0.67.

## Serving

1. Weights = parameters × bytes per parameter (BF16 2, FP8 1, INT4 0.5).
2. Per replica = weights × (1 + headroom, yours, default 30%).
3. GPUs per replica = memory ÷ memory per GPU, rounded up to 1, 2, 4 or 8.
4. Replicas = target tokens/s ÷ measured tokens/s per replica, recorded with the chip it was measured on. Other chips: the figure × (GPUs per replica × memory bandwidth) here ÷ there, an assumption marked to confirm (400 on 2 × H200 is about 221 on 1 × MI300X). Without a measured figure: each chip a starting shape of two nodes with its own GPUs per replica, replica count confirmed against a benchmark on each chip before any quote is final.

## Chip figures (dense BF16, per accelerator)

| Chip | TFLOPS | Memory | Bandwidth | Per node | Source |
|---|---|---|---|---|---|
| H100 SXM | 989 | 80 GB HBM3 | 3.35 TB/s | 8 | https://www.nvidia.com/en-us/data-center/h100/ |
| H200 | 989 | 141 GB HBM3e | 4.8 TB/s | 8 | https://www.nvidia.com/en-us/data-center/h200/ |
| B200 | 2,250 | 180 GB HBM3e | 8 TB/s | 8 | https://www.nvidia.com/en-us/data-center/hgx/ |
| MI300X | 1,307 | 192 GB HBM3 | 5.3 TB/s | 8 | https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html |
| A100 | 312 | 80 GB HBM2e | 2.04 TB/s | 8 | https://www.nvidia.com/en-us/data-center/a100/ |
| L40S | 362 | 48 GB GDDR6 | 0.864 TB/s | 8 | https://www.nvidia.com/en-us/data-center/l40s/ |

An NVIDIA DGX H100 system holds 8 H100 GPUs: the 8-per-node convention used here. https://www.nvidia.com/en-us/data-center/dgx-h100/

## Terms

- 6 weeks = 1,008 h
- 1 month = 720 h
- 2 months = 1,440 h
- 3 months = 2,160 h
- 6 months = 4,380 h
- 1 year = 8,760 h

## Not covered

Checkpointing, failures, data loading and evals (the margin); communication overhead at scale; storage, egress and CPU nodes; price and availability. An estimate is not a quote.