← Back to Benchmarks

Lattice Pulse 2 · retired

Retired from the lineup — the benchmark that taught us a fine-tune can look like a success in production and change nothing. This page remains as the record of that lesson.

Pulse 2 vs base Qwen3-8B

Head-to-head on identity and factual prompts under two conditions: with the Lattice system prompt (what the live site sends) and without it (isolates what the fine-tune itself contributes).

identity without system prompt (Pulse 2 = base)

100%

factual accuracy retained — no forgetting

8/8

identity prompts leaked "Alibaba" / "Qwen" with no prompt

Benchmark chart comparing Lattice Pulse 2 (LoRA on Qwen3-8B) against base Qwen3-8B on identity and factual tasks, with and without the system prompt — **Pulse 2 vs base Qwen3-8B** Identity + factual eval · with & without system prompt

What this shows: the rank-16 LoRA has no measurable effect on identity — without the system prompt, Pulse 2 says "I am Qwen, made by Alibaba" exactly like the base model. The Lattice branding visitors see on the site comes entirely from the system prompt. The genuine positive: the fine-tune caused zero catastrophic forgetting — factual accuracy stayed at 100% across every condition.

Full methodology + per-prompt results: pulse/BENCHMARK_PULSE2.md

Why this matters

Pulse 2 is the cautionary tale that motivated Spark. A LoRA can leave identity completely untouched — the honest test is to strip the system prompt and see what the weights alone say. Spark passes that test; see the comparison.