Mac Studio vs. CMP 170HX: Same 27B Model, Same Settings, Which Serves It Better?

Oct 21, 2026 · 12 min read

We run the same local LLM on two very different machines: a Mac Studio and an Ubuntu VM with a single NVIDIA CMP 170HX (64 GB) passed through to it. Both serve the same model through the same llama.cpp build, with identical launch flags. We wanted to know which one handles real day-to-day use better, and what would change if we moved everything from one to the other.

TL;DR

• Overall, the Mac Studio won 28 of 31 metrics (2 for the CMP 170HX, 1 tie), about 1.55x better by geometric mean.
• For one person chatting, the two feel almost the same. Streaming speed is a tie (about 55 tokens/s on each). The Mac starts replying about half a second sooner.
• The gap grows with concurrency. Two people chatting at once get about 34% faster streaming on the Mac. Four agents get about 45% more throughput and about 23% lower worst-case latency.
• Long prompts are where the Mac pulls ahead. A long prompt (about 17k tokens) processes about 28% faster.
• The CMP 170HX isn't beaten everywhere. Once an agent request is generating, the CMP streams faster in two of our scenarios, though that number swung a lot between runs.
• Chat stalled behind agents on both machines. That comes from sharing two slots, not from the hardware, and it's fixable.
• Software wasn't the reason. We first ran with different llama.cpp builds, then matched them and reran. The CMP box barely changed (1.04x), and the Mac's lead held.
• Bonus: a vLLM server running a different, 4-bit model kept chat responsive (about 0.1 s to first token) even with four agents hitting it. See the bonus section near the end.

The Setup

We matched the configuration on purpose. The Mac server originally ran 8 parallel slots, so we dropped it to 2 to match the CMP box. The CMP box was running an older llama.cpp build (11143), so we rebuilt it at the Mac's exact commit. The CMP 170HX's GPU was idle (0% utilization) at the start, and the benchmark client ran from a third machine (a Mac Mini) so it didn't load either server.

How We Tested

A small Python script (stdlib only) sent streaming requests to each server's OpenAI-compatible endpoint and recorded time to first token, streaming speed, total throughput and latency percentiles. Every scenario ran on both servers with zero errors.

Results

All numbers below are from the matched-build run. Both llama.cpp servers received identical prompts, so their comparison is like for like.

Prompt processing (~17k-token prompt)

Chat

Going from one chatter to two, the CMP 170HX lost about a third of its per-user speed (56 to 38 tok/s). The Mac lost about 9% (55 to 50 tok/s).

The Mac's lead in time to first token for a single user comes from per-request overhead, not prompt speed. A one-token request takes about 0.23 s on the CMP box and about 0.05 s on the Mac, whether the requests are sent back to back or five seconds apart. Network latency isn't the cause (a ping to each server takes under a millisecond), and neither is the GPU waking from idle: the CMP stayed in its top performance state the whole time, and spaced-out requests were no slower than back-to-back ones. The servers' own timers show the extra time is spent inside prompt processing (about 215 to 240 ms on the CMP box versus 45 to 70 ms on the Mac for a trivial prompt), not in the web layer: a health-check request returns in about 3 ms on both. That is far more than a prompt this short should cost at the CMP's measured prompt speed, so it looks like a fixed cost per prompt-processing step. We haven't yet worked out why. (The very first request after a long idle period was slower on both machines: 1.4 s on the CMP versus 0.9 s on the Mac.)

Agents

Agent output looks low on both machines because each request carries a ~6k-token prompt and returns only ~200 tokens, so most of the time goes to prompt processing. That's why the Mac's faster prompt processing shows up so clearly here. Per-stream speed during generation is a different story: the CMP led with 4 agents, the Mac led with 2, and the order flipped between our two runs, so we treat it as a toss-up.

Chat while agents are running

Both machines struggle here. A chat message waited 25 to 45 seconds before the first token, because with only two slots, chat requests queue behind agents. Once chat replies started, the Mac streamed them more than twice as fast (47 versus 22 tok/s).

Does the llama.cpp Build Matter?

Our first run had the CMP box on an older build (11143) than the Mac (11604). To rule that out, we rebuilt the CMP box at the Mac's exact commit and reran.

Across all 31 metrics, the new build was 1.04x better than the old one: 11 metrics up, 6 down, 14 flat. Single-chat and prompt speed barely moved. The metrics that did swing by 40 to 70% were agent per-stream speeds, which also moved in opposite directions in different scenarios. That is run-to-run noise, not a software effect.

What Migrating Would Change

Moving from the CMP 170HX to the Mac Studio:

The one thing that may get worse: agent requests that are already generating sometimes stream slower on the Mac (about 27% slower with 4 agents, about 44% slower in the mixed test). That result was inconsistent between runs, so don't count on it.

Bonus: a vLLM Server Running a Different Model

We also ran the same test against a third machine: a vLLM server running GLM-5.3-Flash (a W4A16 4-bit quant with MTP). It's a different model and a different serving stack, so this is context, not a ranking. Tokens per second can't be compared fairly across different models.

Hardware: four NVIDIA CMP 170HX cards (64 GB each, 256 GB total) running with tensor parallelism across all four, on an AMD EPYC 7502P (32 cores) with 247 GiB of RAM, Ubuntu 26.04.1. That's the same card as our single-GPU test, times four. The build is covered in Four CMP 170HX Cards, One Server.

Server: vLLM 0.32.0rc1.dev215 (a development build), max context 262,144 tokens, up to 32 concurrent sequences (--max-num-seqs 32), 85% GPU memory utilization, and speculative decoding with a separate draft model (7 tokens per step, adjusting down to 5 or 3 as load rises). Prefill is chunked and mixed with decoding.

The long prompt (the same text counts as about 12.5k tokens with GLM's tokenizer) reached its first token in 4.1 seconds, which works out to roughly 3,000 prompt tokens per second. That figure is calculated from the actual prompt token count, since vLLM doesn't report server-side prompt speed. Every scenario ran with zero errors.

The notable result is the mixed test. With four agents running, chat still got its first token in about 0.12 seconds at the median, where the two llama.cpp servers made chat wait 25 to 30 seconds. That supports the slot-isolation point above: vLLM batches requests continuously instead of using a small fixed number of slots, so chat doesn't queue behind agents. This server allows 32 concurrent sequences, where the llama.cpp servers have 2 slots. We also can't separate how much of the result comes from vLLM's scheduling, from using four GPUs, from the speculative-decoding setup, or from this model simply being lighter to run, because we haven't run the same model on both stacks.

Where Each One Makes Sense

What the benchmark shows. With this model, the Mac Studio was equal or better on every speed measure. The CMP 170HX did not win on prompt processing: the Mac processed long prompts about 28% faster (1,038 versus 814 tok/s). The only clean tie is single-user streaming speed (55.7 versus 55.1 tok/s). The CMP led on agent streaming speed in two scenarios, but that result flipped between our two runs, so we don't count it as a win.

What the benchmark doesn't measure. The case for the CMP 170HX rests on things outside our tests:

• Cost. As of October 21, 2026, a CMP 170HX card costs about $2,500. The Mac Studio configuration we tested (M5 Ultra, 36-core CPU, 80-core GPU, 256 GB) starts at $10,799, but the same chip with 96 GB costs $6,799. Our llama.cpp server used about 46 GB of memory with the full 262k context, so the 96 GB model should run this workload comfortably. We expect the same speed from the same chip, but we only tested the 256 GB machine. Against the 96 GB price, the Mac costs about 2.7x as much as one card for about 1.55x the performance (about 4.3x at the 256 GB price). The comparison isn't like for like, though: the card needs a host system around it, which we haven't priced, and four cards ($10,000) cost more than the 96 GB Mac. If one setup costs much less, a 1.5x speed gap can still favor it.
• Power. The CMP's board power in our samples was about 44 W idle and roughly 52 to 140 W in short bursts. We didn't measure sustained load, and we didn't measure the Mac's draw.
• Adding cards. Our 4-card vLLM bonus kept chat responsive while agents ran, but that was a different model and a different stack, so it doesn't tell us how this model would scale on more cards.
• Software. The CMP box runs CUDA tools such as vLLM and multi-GPU tensor parallelism. We didn't test the Mac with anything beyond llama.cpp.
• Sharing. The CMP VM holds the 27B model and a second model (Gemma 12B) in its 64 GB at the same time.

A rule of thumb. If you want one machine to serve agents or several people, the Mac Studio is the better pick on speed. If you already own the CMP hardware, or price per unit matters, a single CMP is fine for one or two people chatting: same streaming speed, about half a second slower to start. Long prompts and heavy agent traffic are where it falls behind.

Takeaways

1. If it's just one or two people chatting, either machine is fine. The Mac is a bit snappier, but you'd need to be watching the clock.
2. If agents are in the picture, the Mac is the better home. Agent prompts are long, and prompt processing is where it leads.
3. Neither machine handles chat and agents sharing two slots well. If interactive chat matters, the fix is slot isolation. Reserving slots for chat, or running a separate instance for agents, would likely help far more than any hardware swap. We haven't tested this yet.
4. The Mac Studio wins every speed measure, but speed isn't the only factor. The CMP 170HX matches it on single-user streaming and runs the whole 27B Q8 model on one card. Price, power and room to add cards, which we didn't measure, decide whether the CMP still makes sense. See "Where each one makes sense."

Caveats

• One model, one quantization, one run per scenario (plus one repeat on the CMP box). The two machines also differ in platform, not just silicon. The CMP 170HX box runs inside a Proxmox VM, so any virtualization overhead is part of its numbers.
• The agent and mixed scenarios use small samples (8 to 16 requests), so treat differences there as directional. Single-chat and prompt-processing results were consistent across both runs.
• The vLLM bonus uses a different model, a different quantization, four GPUs instead of one, and a different serving stack, so it can't be compared directly with the Qwen results.
• Streaming speed and time to first token are measured at the client over the LAN.
• We matched flags on purpose. Tuning each machine separately (batch sizes, slot counts, KV cache type) could shift the results.

Notes on the Model and Setup

• The model. Huihui-Qwen3.8-27B-abliterated is a community variant of Qwen3.8 27B (about 27.3 billion parameters) in which the model's built-in refusal behavior has been removed ("abliterated"). We used it because it's the model we run day to day. Removing refusals shouldn't change speed, so the benchmark results should carry over to the standard Qwen3.8 27B at the same quantization.
• The quantization. Q8_0 is an 8-bit GGUF file of about 29 GB. It's close to full quality, which is why it needs a lot of memory and why the 64 GB card and the 256 GB Mac both handle it comfortably.
• Vision. Both llama.cpp servers also load the model's vision projector, so image input was available. The benchmark only sent text.
• Reasoning and speculative decoding. Both servers ran with reasoning effort set to low and the same speculative-decoding setting (MTP drafting, 2 tokens). Both affect speed, which is why we kept them identical.
• The bonus model. GLM-5.3-Flash is a different model entirely, served from a 4-bit (W4A16) quant with its own draft model for speculative decoding. We included it only to show how a different serving stack behaves under the same load.
• The test script. The benchmark is a single stdlib-only Python script that works with any OpenAI-compatible server. We can publish it if readers want to reproduce the results.

Related: Mac Studio vs. CMP 170HX for image generation and Four CMP 170HX Cards, One Server.