Sizing method
How the sizing sheet works
Every line on the run desk, with its formula, its inputs and where each constant comes from. Utilisation, margin, headroom and LoRA fraction are your assumptions; you set them on the sheet.
Operated by Tempered Technologies LLC, 8 The Green STE B, Dover, DE 19901.
Training and fine-tuning
- Compute = 6 × N × D. N is parameters, D is training tokens; about 6 FLOPs per parameter per token for the forward and backward pass (Kaplan et al., 2020). Example: 6 × 13 B × 300 B = 2.34 × 10²² FLOPs. If you give no token count, the sheet uses 20 tokens per parameter (Hoffmann et al., 2022) and marks it not stated.
- Effective per GPU = peak dense BF16 × utilisation. Utilisation is yours, 40% until you change it. H100 SXM: 989 TFLOPS × 40% = 396 TFLOPS.
- GPU-hours = compute ÷ effective ÷ 3,600 × (1 + margin). The margin is yours, 15% until you change it, and covers what the formula leaves out (below). Example: about 18,900 GPU-hours.
- GPUs for the window = GPU-hours ÷ window hours, rounded up to whole nodes of 8, dividing the GPU-hours as printed on the sheet. Six weeks is 1,008 h: 18.75 GPUs, so 24 GPUs, 3 nodes.
- With a start window, the run must fit from the latest start. "Start any day 4 to 15 January, done by 15 February" leaves 31 days (744 h) if the provider starts on 15 January: 25.40 GPUs, so 32, 4 nodes. When the deadline was given as a duration ("done in 6 weeks"), the desk asks whether it holds whatever the start or counts from whichever day it starts; until you answer it sizes for the first and sends the question with the figure for the second (24 GPUs here). The plot draws the run from the latest start, in the conflict colour if it would end late.
- Wall-clock = GPU-hours ÷ GPUs. At 24 GPUs, about 787 h, 4.7 weeks.
- Reserved = GPUs × term hours. On a 6-week term, 24 × 1,008 h = 24,192 GPU-hours; the run uses 78% of it. The shortest term you asked to quote that covers the run is the one drawn.
LoRA. Frozen weights skip their own gradients, so a LoRA fine-tune is a fraction of full fine-tune compute. The fraction is your assumption, 0.67 until you change it (about 4 of the 6 FLOPs per parameter per token), and it is shown on the sheet.
Known GPU-hours. If you already know the work in GPU-hours, the first three lines are your figure and the sheet starts at the count.
Serving
- Weights = parameters × bytes per parameter: 2 for BF16, 1 for FP8, 0.5 for INT4. 70 B at BF16 is 140 GB.
- Per replica = weights × (1 + headroom) for KV cache and activations. Headroom is yours, 30% until you change it: 182 GB.
- GPUs per replica = memory ÷ memory per GPU, rounded up to a tensor-parallel width of 1, 2, 4 or 8. 182 GB on H200 (141 GB) is 2; on H100 (80 GB) it is 4.
- Replicas = target tokens/s ÷ your measured tokens/s per replica. Throughput depends on context length, batch and serving stack, so the sheet uses only a figure you measured, and records the chip you measured it on. Without one, each chip is a starting shape of two nodes, so one can drain without stopping service, with its own GPUs per replica, and the replica count is confirmed against a benchmark on each chip before any quote is final.
- Other chips, from your measured figure: × (GPUs per replica × memory bandwidth) here ÷ there. Token generation is bound by memory bandwidth, so this is the stated assumption for carrying your figure to a chip you did not measure, and it is marked to confirm. Example: 400 tokens/s on a 2 × H200 replica (2 × 4.8 TB/s) gives about 221 tokens/s on a 1 × MI300X replica (5.3 TB/s): 5,000 tokens/s takes 23 replicas, 24 GPUs, against 13 replicas, 32 GPUs, on H200.
Chip figures
Dense BF16 tensor throughput and memory per accelerator, from each vendor's datasheet. Vendors often list BF16 with sparsity, which is twice the dense figure; the sheet uses dense.
| Chip | Dense BF16, TFLOPS | Memory | Bandwidth | Per node | Source |
|---|---|---|---|---|---|
| H100 SXM | 989 | 80 GB HBM3 | 3.35 TB/s | 8 | nvidia.com, H100BF16 tensor 1,979 TFLOPS is listed with sparsity; dense is half. |
| H200 | 989 | 141 GB HBM3e | 4.8 TB/s | 8 | nvidia.com, H200Same compute as H100 SXM; more and faster memory. |
| B200 | 2,250 | 180 GB HBM3e | 8 TB/s | 8 | nvidia.com, HGX B200HGX B200 lists 36 PFLOPS BF16 with sparsity and 1,440 GB across 8 GPUs; per GPU dense is 2,250 TFLOPS and 180 GB. |
| MI300X | 1,307 | 192 GB HBM3 | 5.3 TB/s | 8 | amd.com, MI300XPeak BF16 1,307.4 TFLOPS dense. |
| A100 | 312 | 80 GB HBM2e | 2.04 TB/s | 8 | nvidia.com, A100BF16 tensor 624 TFLOPS is listed with sparsity; dense is half. |
| L40S | 362 | 48 GB GDDR6 | 0.864 TB/s | 8 | nvidia.com, L40SBF16 tensor 733 TFLOPS is listed with sparsity; dense is 362. PCIe, no NVLink between cards. |
An NVIDIA DGX H100 system holds 8 H100 GPUs: the 8-per-node convention used here. nvidia.com, DGX H100.
Term conventions
6 weeks = 1,008 h; 1 month = 720 h; 2 months = 1,440 h; 3 months = 2,160 h; 6 months = 4,380 h; 1 year = 8,760 h.
What the estimate does not cover
- Checkpointing, restarts after hardware failures, data loading stalls and evaluation runs: the schedule margin is there for these.
- Communication overhead at large node counts, pipeline bubbles and attention cost at long context, which lower real utilisation; set utilisation from your own runs if you have them.
- Storage, egress and CPU nodes; ask for them in the request and they are quoted with the cluster.
- Price and availability. An estimate is not a quote; providers confirm fit, dates and price.