Quark on your Mac: the MLX build is here

Releases Aug 19, 2026

Up until now Quark only lived on a server. The weights were a raw PyTorch checkpoint, which is fine for the site chat but hopeless for your laptop. No dramas anymore: we've shipped a 4-bit MLX build that runs fully on-device at around 100 tokens a second on Apple Silicon, no cloud involved. Here's how the conversion worked and how to run it yourself.

The problem

Quark's checkpoint is a single 4.3GB PyTorch file with 189 tensors and 1.49B parameters, saved straight from the training loop. It runs fine on the Azure VM that powers the site chat, but there's no clean way to run a raw checkpoint like that on a Mac. You would need the full training stack, a mountain of RAM, and a lot of patience.

The fix is MLX, Apple's array framework for Apple Silicon. MLX gives you the whole model on the GPU with a fraction of the memory, which is exactly what a 1.5B model needs to run properly on a laptop.

The conversion

The plan was simple in theory: load the checkpoint, quantise the big matrices to 4-bit, and write everything back out as MLX weights. The transformer linears (attention projections, MLPs, the logit head) went to 4-bit with a group size of 64, which is the convention for MLX models. The token embedding and the value embeddings stayed in bf16.

That last call mattered more than you'd think. Quark's value embeddings are a genuinely big part of the model: 13 full-width (32768 × 1536) tables, 654M parameters all up, on alternating layers. They're additive priors gated straight into the attention stream, so we kept them at full precision rather than risk the noise. The final build is 1.7GB, which is the honest trade between size and quality.

A few things surprised us while we were in there. The vocab needed no padding at all (32768 is already a clean number, so the checkpoint's own padding logic was a no-op). Every norm in the model is weightless, which is a genuine quirk of the design and saves a chunk of space. And the custom GPT-4 style tokenizer had to be dumped by hand into a JSON file, because the usual tooling assumes the GPT-2 byte regex and would silently mangle Quark's vocabulary.

How it runs

The proof is in the numbers. On a MacBook Air, the 4-bit build does roughly 70 to 100 tokens a second with greedy decoding, with the prompt prefill taking a fraction of a second. Multi-turn chat keeps the whole conversation in one KV cache, so the model genuinely remembers what you said two turns ago, no tricks. And because everything runs through MLX on the Metal GPU, your CPU is free to do other things while Quark chats away.

The good bit for privacy nerds: nothing leaves your machine. The model is local, the weights are local, the conversation is local. We've effectively got Quark running on the couch, which is where small open models belong.

How to run it

Grab the weights from Hugging Face (model type nanochat2, 4-bit MLX), then run the bundled terminal runner:

python run_quark_terminal.py --model lattice-quark-1.5b-mlx --prompt "G'day Quark, what are you?"

Omit --prompt for interactive chat. Quark speaks the SFT marker format from training: <|bos|><|user_start|>...<|user_end|><|assistant_start|>, which the runner handles for you. Use --cpu if you ever want it off the GPU entirely.

The honest version

Greedy only for now. The runner decodes greedily. Sampling and temperature are on the list, just not in the first release.
The model type is ours. We ship Quark as nanochat2 because the nanochat type floating around in the standard MLX tooling is an incomplete port (no value embeddings, no smear, no backout). Loading Quark into that port would silently produce garbage, so we named ours differently and wrote the complete one ourselves.
App support is coming. A full Swift port is ready for our own Mesh app, and tooling integration (Locally AI and friends) is the next frontier. The terminal runner is the honest first step, and it already does the whole job.

What this means

Quark started as "can we train a real model from nothing". Now it's "can we run it on a laptop". Both answers turned out to be yes, and the gap between them was mostly paperwork: a quantiser, a tokenizer dump, and a bit of patience with shape errors. The weights are open on Hugging Face if you want to have a crack yourself.