Same Model, Two Machines: Do the Answers Match?

Oct 24, 2026 · Reclaimed Node · 8 min read

The short version

The same 27-billion-parameter model gave nearly identical answers on a Mac Studio and on a server built around CMP 170HX GPUs: scores landed within two points of each other on four of the five tests we ran, and the fifth, long context, tied once we only asked whether the right answers appeared. All of that is inside the margin of error for the number of questions asked.

We also found that one setting matters far more than the hardware. With a 2,048-token reply limit, about one answer in five was cut off before it finished, and the model looked 8 to 14 points worse on MMLU-Pro and MATH-500 than it really is. Raising the limit to 6,144 tokens fixed most of it.

The setup

Both machines served the same model file with the same llama.cpp build and the same start-up flags, so the only thing that differs is the hardware underneath.

The flags on both were -ngl 999 -c 262144 -fa on --jinja --parallel 2 --spec-type draft-mtp --spec-draft-n-max 2. That means every layer on the GPU, flash attention on, and speculative decoding using the model's own multi-token-prediction head to draft two tokens at a time. The server's default reasoning effort was set to low, which only matters when thinking is on. It was off for every test here.

The tests ran from a third computer, a Mac Mini, over the local network. It sent both machines the same questions at the same moment, at temperature 0, two at a time per machine, and scored the replies itself.

How the answers are scored

Every answer is checked by code, not by another AI. The questions come unchanged from published tests, so anyone can check our work.

• IFEval (541 questions) gives instructions like "write at least 300 words" or "use no commas", and checks each one mechanically.
• GSM8K (1,319) is grade-school maths; the final number must match.
• MMLU-Pro (12,032) is multiple-choice knowledge across 14 subjects, with ten options per question.
• MATH-500 (500) is competition maths; the boxed final answer must match after normalising.
• Long context is generated by this tool: a fact hidden in 4K to 32K tokens of filler, then the model is asked to find it.

We report two scores. Strict accepts only the answer in the requested format. Loose is more forgiving about format, such as an answer that is right but not boxed, or, in the long-context test, a reply that contains the right answer along with extra names. A big gap between them means the answer was in the reply but not in the form asked. Cut off counts replies that hit the token limit before finishing, because a cut-off answer usually scores zero.

Results

On the four main tests the two machines are never more than two points apart, and long-context retrieval ties at 100% when we only check that the right answers appear. Each cell shows strict / loose score, then replies that were cut off.

Each test used the same evenly spread sample of questions on both machines, with thinking off and temperature 0. Replies are not identical word for word. The two backends do arithmetic slightly differently, and a tiny difference can change which word the model picks next, so the machines occasionally disagree on a single question. On the long-context test, which has short replies, 27 of 28 were word-for-word identical. On MATH-500, where replies run for paragraphs, only 34 of 100 were identical, yet the two machines agreed on right or wrong for 96 of 100 questions. The disagreements go both ways and cancel out.

The setting that mattered most: reply length

Raising the reply limit from 2,048 to 6,144 tokens lifted MMLU-Pro by 8 to 14 points and MATH-500 by 14 points, on both machines. The model had the answers; it ran out of room to write them.

Even with thinking switched off, this model writes long step-by-step answers. The effect shows most in subjects that need long calculations: MMLU-Pro engineering went from 0 of 8 right on the Mac Studio to 3 of 8.

A few replies still hit 6,144 tokens. We read the four MATH-500 ones on each machine: all were hard problems in intermediate algebra, geometry or precalculus where the model was still working through the solution, with no repeating loops. Three of the four were the same questions on both machines. We did not read the MMLU-Pro cut-offs. Treat both scores as a floor.

The practical lesson: when you see a benchmark score for any local model, check the reply limit. A model that scores 68% with one limit and 82% with another has not changed, only the test setup has.

Long context: finding facts up to 32K tokens

Both machines found every needle and every multi-key fact, from 4K up to 32K tokens of filler, and wrote word-for-word identical replies to 27 of the 28 questions. The misses are all in the variable chain, where the model found the whole chain but often added names that did not belong, and they are nearly the same on both.

The three tasks are a single fact hidden at 10%, 50% and 90% of the text (needle), one correct value among ten look-alike keys (multi-key), and a four-step chain of variable assignments with a second, distracting chain (variable chain). The table shows correct answers out of the number asked, Mac Studio first, then CMP 170HX.

On the loose score, which only asks whether the right answer appears in the reply, both machines got 28 of 28. Strict also rejects a reply that lists a variable from the distractor chain, and that is what happened: the correct answer is four variable names, and the replies that failed listed six or seven, adding two or three names that hold a different value. That is a real mistake in precision, not a formatting slip. The Mac Studio made it on five of its eight variable-chain questions and the CMP server on four. With two questions per cell we would not read a trend by size, and a one-question gap between machines is noise. This test is made by LLM-Super-Bench in the style of RULER and needle-in-a-haystack, so its numbers cannot be compared with those benchmarks' published scores.

What these numbers do and do not show

• Sample size. 100 questions gives a margin of error of about 3 points at a 97% score and 8 points at 80% (95% confidence). A 2-point gap between machines is noise. Per-subject results, which have 3 to 11 questions each, are noisier still.
• Not leaderboard scores. We ask each question once with a plain prompt and no worked examples. Published leaderboards usually use few-shot prompts and their own harnesses, so these numbers are comparable between our two machines, not with a chart from a model card.
• One specific build. The model is a Huihui abliterated fine-tune of Qwen3.8 27B, not the official release, and the scores are that build's.
• Thinking was off. Results with thinking on would differ and need much larger reply limits.
• IFEval checks are ports. We ported Google's IFEval checkers to TypeScript. The language-detection checks are approximate, and sentence counting is simple.
• Speed is a separate question. This post is about whether the answers match. For tokens per second, see the earlier head-to-head post.
• Speculative decoding was on for both machines. At temperature 0 it should not change the answers, but it is part of the setup.

Run it yourself

The tests are part of LLM-Super-Bench, free and open source. Add your servers to race.config.json, then:

bun run update                 # get the latest version
bun run quality:fetch all      # download the published question sets once
for s in ifeval gsm8k mmlu-pro math500; do
  bun run quality --suite $s --limit 100 --streams 2 --max-tokens 6144 --thinking off
done
bun run quality --suite longctx --sizes 4k,8k,16k,32k --streams 1

Set --streams to the number of parallel slots your server has. Each run prints which machines it is asking, the scores by subject, and how many replies were cut off, and saves everything to a results file.

Credits and licences

The question sets are the work of their authors and are used unchanged under their licences:

• IFEval, Google Research (Apache 2.0)
• MMLU-Pro, TIGER-Lab (MIT)
• GSM8K, OpenAI (MIT)
• MATH-500, from OpenAI's PRM800K split of the MATH dataset by Hendrycks et al.
• The long-context test is original to LLM-Super-Bench, modelled on NVIDIA's RULER (Apache 2.0) and Greg Kamradt's needle-in-a-haystack test (MIT).

LLM-Super-Bench grew out of NetworkChuck's harness project (MIT).

Related: Mac Studio vs. CMP 170HX: Same 27B Model, Same Settings · Race Your Local AI Machines and Trust the Numbers · llama.cpp vs MLX on an M5 Ultra Mac Studio.