Lattice
Lattice Mini: a 42M model built from scratch
Most small models are small fine-tunes — a stamp on someone else's pretrained brain. Mini is the other kind: 42M parameters, built from nothing, with no borrowed parts.
What Mini is
Lattice Mini is a 42M-parameter GPT decoder trained from scratch — the smallest model in the Lattice lineup, and the one that made no assumptions. No borrowed tokenizer, no pre-trained weights, no head start.
| Spec | Value |
|---|---|
| Architecture | 12-layer GPT decoder |
| Params | ~42M |
| Context | 512 tokens |
| Tokenizer | Custom ~8k BPE (trained on the corpus) |
| Pretraining | WikiText-2 |
| Instruction tuning | Alpaca (chat format) |
The pipeline
Nothing about Mini was handed to us. The tokenizer was trained on the corpus it would read. The model was pretrained to predict the next token. Only then was it instruction-tuned into something that can hold a conversation at all. Three separate stages, each one a place the training could silently fail.
From scratch vs fine-tuning: different jobs
It's tempting to think of fine-tuning and from-scratch training as the same thing at different speeds. They're not. They answer different questions:
From-scratch asks "can this model learn to write at all?" A 42M model has to acquire vocabulary, grammar, world knowledge, and reasoning from raw text — and it has far fewer weights to store that in. Mini's pretraining ran for tens of thousands of steps and still produces answers that are "often wrong or repetitive", in the space's own words.
| Spark 1.5B (fine-tune) | Mini 42M (from scratch) | |
|---|---|---|
| Starting point | Qwen2.5-1.5B, pre-trained | Random weights |
| What it learns | Identity + chat style | Language itself |
| Where knowledge lives | Inherited from the base | Earned from the data |
| Result | Fluent, knows it's Lattice | Barely talkable, honest about it |
The distinction isn't a speed thing — it's an ownership thing. Spark is fluent because Qwen's makers taught a base model to write English; we stamped our name on it. Mini is barely talkable because it had to buy every word with its own weights. Neither approach is "better" — they're different jobs. But if you want to have paid the pretraining bill yourself, from-scratch is the only way.
What Mini is good for
Honest expectations: Mini is a research artifact, not a product. It demonstrates the full pipeline on hardware that fits in a backpack, and it makes the failure modes visible — wrong answers, repetition, confident nonsense. That visibility is the point at this scale.
Mini · "What is the capital of France?" Expect something vaguely Paris-adjacent, or something entirely made up. Lower temperature helps; patience helps more.