GPT-124M from scratch: benchmarking and restart recovery
A randomly initialized GPT-124M trained on 1,000,341,504 tokens using free Kaggle T4s, with verified cross-session recovery and final in-domain evaluation. A completed pretraining demo, not an instruction-tuned assistant.
Pilot, recovery and completion
Recorded milestones · the single-T4 pilot is a separate run.
Pilot: 6,881,280 total tokens; 6,750,208 inside 904.14 seconds. Timing boundary
- Saved checkpoint548,405,248 tokens
548,405,248 - Completed run1,000,341,504 tokens
1,000,341,504
Scale: 1,000,000,000-token target. The final count includes the recorded fixed-batch overshoot. These are saved milestones, not live progress.
State that crossed the session boundary
- model
- optimizer
- scheduler
- scaler
- RNG state
- data offset
What the replay verified
The full-state replay test restored the model, optimizer, scheduler, scaler, RNG state, and data offset. The tested continuation reproduced the expected result exactly.
Fresh-session continuation
The full-state checkpoint's hash was verified. In a fresh successor session, the distributed replay check passed, and training then advanced beyond the saved checkpoint. This tests a stronger boundary than keeping one process alive: saved state was recovered in a new two-GPU session and used to continue the run.
Recovery limits
These results establish the tested replay and saved-session recovery cases. They do not establish identical behavior across every hardware configuration or every possible interruption.
What completion means
The output remains repetitive. This is a next-token pretrained base model, not an instruction-tuned conversational assistant. The held-out figures come from the in-domain FineWeb-Edu document-hash split described above, not a general capability benchmark or a fully decontaminated evaluation.
I trained a small GPT model on free Kaggle GPUs. The work made training measurable and restartable: a timed single-GPU pilot, a checkpoint that preserves the training state, and continuation of a longer two-GPU run across fresh sessions.
This record was first published on October 5, 2026. The earlier session milestone was verified on October 6 in Asia/Shanghai (October 5 UTC). Final completion was verified at 01:46 UTC on October 6, 2026: the one-billion-token demo and its in-domain held-out evaluation are complete. The earlier measurements below remain as the run history.
Try the model
Give it the beginning of a passage to continue. This is a base model, not an instruction-tuned chat assistant; its completions can be repetitive.
What starts from scratch
The model has 124,439,808 randomly initialized parameters. Its GPT-2 architecture uses 12 layers, 12 attention heads, a width of 768, and a 1,024-token context. No pretrained model weights are loaded.
I use the existing GPT-2 tokenizer and public FineWeb-Edu data. Starting the model weights from scratch does not mean inventing a new tokenizer or collecting a new corpus.
A pilot with an explicit timing boundary
The single-T4 pilot processed 6,881,280 tokens in total. Its timed portion covered 6,750,208 tokens in 904.14 seconds, including checkpoint work. That gives approximately 7,465.9 tokens per second. Reported peak GPU memory was 4.46 GiB.
| Single-T4 measurement | Result |
|---|---|
| Total pilot tokens | 6,881,280 |
| Tokens inside the timing window | 6,750,208 |
| Timed duration, including checkpoint work | 904.14 seconds |
| Throughput inside that window | 7,465.9 tokens/second |
| Reported peak GPU memory | 4.46 GiB |
The timing window matters. Dividing the total pilot token count by a duration that measured only part of the run would produce a different and misleading rate.
On a held-out set of 65,536 tokens, loss fell from approximately 10.99 to 8.41. Generated samples were still repetitive. These are early diagnostics, not a capability evaluation or evidence that the model is ready for practical use.
A checkpoint must preserve the next step
Saving model weights alone is not enough to resume the same training trajectory. The optimizer, learning-rate schedule, mixed-precision scaler, random-number generators, and data position all influence what happens next.
The full-state replay test restored the model, optimizer, scheduler, scaler, RNG state, and data offset. The tested continuation reproduced the expected result exactly.
I then tested a separate failure boundary: a completely fresh Kaggle session loaded the saved checkpoint and resumed successfully. That matters because a checkpoint that works only while the original process is alive has not demonstrated useful recovery.
These results establish the tested replay and saved-session recovery cases. They do not establish identical behavior across every hardware configuration or every possible interruption.
The longer two-T4 run
A separate training token store contains 1.1 billion token positions. A document-hash holdout separates the selected evaluation documents from training. No additional near-duplicate decontamination has been established, so this split should not be described as a fully decontaminated benchmark.
An initial two-T4 distributed-data-parallel test reached approximately 14,787 tokens per second across the two GPUs. That is an initial measurement, not an established sustained rate for the longer run.
A completed session and a verified continuation
At the verified October 5 UTC session boundary, the first production session had safely finished approximately 10.5 hours of two-T4 training and saved its full state. The saved checkpoint represented 548,405,248 processed tokens: 54.84% of the one-billion-token target. This is a fixed checkpoint milestone, not a live progress counter.
| Two-T4 production-session measurement | Result |
|---|---|
| Tokens at the saved checkpoint | 548,405,248 |
| Share of the one-billion-token target | 54.84% |
| Reported sustained throughput | Approximately 14,533 tokens/second |
| In-domain held-out loss | 4.0352 |
| In-domain held-out perplexity | 56.55 |
The full-state checkpoint's hash was verified. In a fresh successor session, the distributed replay check passed, and training then advanced beyond the saved checkpoint. This tests a stronger boundary than keeping one process alive: saved state was recovered in a new two-GPU session and used to continue the run.
These production figures are separate from the single-T4 pilot above. The held-out loss and perplexity describe the selected in-domain holdout, with the same limits on decontamination already noted; they are not a general capability score.
The one-billion-token run was not complete at that intermediate verification point. The final result below follows the recovered continuation.
Completed one-billion-token demo
Final verification at 01:46 UTC on October 6 confirmed 1,000,341,504 predicted training tokens. The slight overshoot of one billion comes from finishing a fixed global batch. The model still has 124,439,808 parameters initialized from scratch; no pretrained weights were introduced.
| Final two-T4 result | Measurement |
|---|---|
| Predicted training tokens | 1,000,341,504 |
| In-domain held-out loss | 3.5953 |
| In-domain held-out perplexity | 36.43 |
| Held-out token predictions evaluated | 65,536 |
| Final training-batch loss | 3.6821 |
| Sustained continuation throughput | Approximately 15,683 tokens/second |
The final training-batch loss is distinct from the held-out evaluation loss. The continuation throughput describes the resumed session, not the single-T4 pilot or the earlier two-T4 session.
The final full-state checkpoint is approximately 1.65 GB. Its SHA-256 checksum was verified, and its saved token count and data offset both match 1,000,341,504. Together with the earlier fresh-session numerical replay, this establishes the tested recovery path and a verified final saved state.
The run used free Kaggle resources only. This completes the one-billion-token demonstration; it does not announce or commit to further training.
What completion does—and does not—mean
The output remains repetitive. This is a next-token pretrained base model, not an instruction-tuned conversational assistant. The held-out figures come from the in-domain FineWeb-Edu document-hash split described above, not a general capability benchmark or a fully decontaminated evaluation.
The completed result is a small model trained from random initialization, measured through a single-GPU pilot and two-GPU sessions, recovered across a fresh session, and saved with verified final counters and checksum. Better loss and a completed token budget do not by themselves establish usefulness or reliable reasoning.