Running qwen3.6:35b on dual Tesla P40s with Ollama

I spent a benchmarking session getting qwen3.6:35b running well on an older dual Tesla P40 machine with Ollama. The goal was practical: make it useful for coding, understand which runtime knobs matter, and keep a correctness benchmark next to the throughput numbers.

The short version: the normal qwen3.6:35b tag on Ollama 0.30.10, cuda_v12, one active request, and locked clocks produced about 48.4 output tokens per second. For larger coding prompts, q8_0 KV cache with flash attention was the best setting I tested. Prompt-cache discipline mattered more for latency than any kernel toggle. MTP helped only for predictable code-completion or function-generation prompts, and should stay off for open-ended planning or prose.

Table of contents

• Results at a glance
• Hardware and runtime setup
• Measurement method
• Correctness benchmarks
• Context length and KV cache
• Prompt-cache results
• MTP and speculative decoding
• Settings that did not help much
• Research ideas not tested here
• Recommended setup
• Takeaways

Results at a glance

The main results are easier to read as decisions than as a large summary table. These are the settings I would carry forward from this round of testing:

• Use the normal qwen3.6:35b tag with Ollama 0.30.10, cuda_v12, one loaded model, and one active request. The stable short-prompt operating point was about 48.4 output tok/s with comfortable temperatures.
• For coding context windows, prefer q8_0 KV cache with flash attention. Allocating 8k or 16k context did not hurt short prompts, but actually filling those contexts slowed decode; q8 KV was the best long-context setting tested.
• Keep prompts cache-friendly. Repeating a 7011-token coding prompt dropped total wall time from 21.54 s to about 2.8 s when the long prefix stayed stable.
• Keep the correctness guardrail next to every speed result. The rapid suite measured 10/12 pass@1 on the baseline and 11/12 with q8 KV; the harder supplement measured 6/12 and is useful once easy practical tasks are saturated.
• Route MTP by prompt shape. Draft 2 helped predictable code completion and BigCodeBench-style function generation, but MTP was slower for open-ended prose/planning. Draft 4 did not add meaningful speed over draft 2 on the easy practical subset and was worse on the harder subset in this run.
• Treat greedy decoding, MMQ/cuBLAS forcing, NUMA pinning, and two parallel requests as non-wins for this setup. They either did not move throughput materially or made the interactive case worse.

Hardware and runtime setup

The machine used two Tesla P40s. The useful model tags in this experiment were:

The final steady short-prompt setup used Ollama 0.30.10, cuda_v12, one loaded model, one active request, and spread scheduling across the two cards.

Representative Ollama service settings:

[Service]
Environment="OLLAMA_LLM_LIBRARY=cuda_v12"
Environment="OLLAMA_CONTEXT_LENGTH=4096"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_LOAD_TIMEOUT=10m"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_SCHED_SPREAD=1"

The P40s supported application clocks, so I enabled persistence mode and locked clocks before the confirmation run. This mostly improved repeatability; it did not turn the machine into a different throughput class.

Clock-locking commands:

sudo nvidia-smi -i 0,1 -pm 1
sudo nvidia-smi -i 0,1 -ac 3615,1531

Measurement method

I measured throughput through Ollama's non-streaming generate API. Output throughput came from the timing fields returned by the API, not from wall-clock parsing of streamed text.

Throughput calculation:

eval_tps = eval_count / (eval_duration_ns / 1e9)
prompt_tps = prompt_eval_count / (prompt_eval_duration_ns / 1e9)

For coding quality, I used execution-based tests rather than relying on throughput alone. A faster setting is not useful for coding if it causes more parse failures, wrong programs, or timeouts.

Representative Ollama generate call:

curl http://127.0.0.1:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6:35b",
    "prompt": "Write a focused implementation plan for ...",
    "stream": false,
    "options": {
      "num_ctx": 8192,
      "num_predict": 512,
      "temperature": 1.0,
      "top_k": 20,
      "top_p": 0.95
    }
  }'

Correctness benchmarks

I used newer execution-based coding benchmarks as the main guardrail:

• LiveCodeBench for recent contest-style code generation. The selected tasks came from January through March 2025.
• BigCodeBench for practical Python tasks involving standard-library and API use.

The rapid suite used 12 tasks: six LiveCodeBench tasks and six BigCodeBench tasks. With the normal service baseline, greedy decoding, num_ctx=8192, and num_predict=2048, it measured 10/12 pass@1.

The two rapid-suite failures were useful. One generated a long solution that called an undefined helper. Another hit the 2048-token cap and failed the first hidden test. That gave the suite enough headroom to catch regressions while still running quickly.

Rapid-suite task list and baseline results:

I then reran the runtime variants against the rapid suite:

Because the first BigCodeBench mini subset was saturated at 6/6, I added a harder 12-task supplement. It used six harder LiveCodeBench tasks and six BigCodeBench-Hard tasks. This was not meant to replace the rapid suite; it is a second-stage check for settings that already look good.

Harder task IDs:

LiveCodeBench: abc387_f, abc388_e, abc389_e, abc390_d, abc397_d, abc398_f.

BigCodeBench-Hard: BigCodeBench/287, BigCodeBench/313, BigCodeBench/454, BigCodeBench/777, BigCodeBench/928, BigCodeBench/990.

Context length and KV cache

For short prompts, simply allocating a larger context window did not reduce decode throughput. The 4k, 8k, and 16k tests all landed around 48.18 tok/s:

The relevant coding case is different: the prompt actually fills most of the context and leaves room for generated output. With f16 KV, decode slowed as the prompt grew:

I then reran calibrated long prompts with KV-cache variants. The harness warmed the same long prompt before the measured request, so output tok/s is the main comparison.

q8_0 KV cache was the best practical long-context decode setting: 43.19 tok/s at 8k and 41.09 tok/s at 16k. q4_0 saved more KV memory but was slower. The quantized-KV runs also had flash attention enabled, so this is best read as a practical Ollama setting result rather than a pure isolated KV-precision experiment.

For coding use on this box, 8k is the better default. Use 16k when the extra repository context is valuable enough to accept slower output. Leave room for output tokens; an early test filled the entire context and left almost no generation budget.

Prompt-cache results

The largest latency improvement came from not recomputing the same long prompt. I ran four identical 8k-context coding prompts against one server, using q8_0 KV cache and no same-prompt warmup.

The API reported the full prompt token count on every request, but the server log showed that only a few prompt tokens were evaluated after the first request. Decode speed did not change much because output tokens still had to be generated normally. The win was latency: repeated requests dropped from about 21.5 seconds to about 2.8 seconds.

I also tested this against real execution-checked tasks with a long stable repository-like prefix:

The practical rule is simple: put stable repository context first and keep it byte-for-byte stable across turns. Put changing task instructions, diffs, and one-off details at the end. Avoid timestamps, random IDs, or whitespace churn near the start of the prompt when they do not matter.

MTP and speculative decoding

I tested the explicit MTP tag, qwen3.6:35b-a3b-mtp-q4_K_M. Ollama logs showed draft-MTP speculative decoding was active.

MTP is supposed to improve throughput by drafting multiple future tokens and asking the target model to verify them. When the target accepts several drafted tokens, the system advances multiple output positions for roughly one verification step plus a cheaper draft step.

The simplified MTP loop:

normal decoding:
  target pass -> token 1
  target pass -> token 2
  target pass -> token 3
  target pass -> token 4

MTP good case:
  draft path proposes token 1, token 2, token 3, token 4
  target model verifies the candidate sequence
  accepted prefix advances the output by several tokens

The speedup depends on high acceptance and low draft overhead. It is most likely to help on low-entropy continuations: boilerplate code, repeated structure, syntax-heavy completions, and fixed schemas. It can hurt when the prompt is open-ended and the draft path is often wrong.

On an open-ended technical checklist prompt, MTP was slower than the normal tag:

In the draft-4 run, position acceptance fell quickly: about 57.5% for the first draft token, 25.3% for the second, 9.6% for the third, and 3.1% for the fourth. The system generated too many rejected draft tokens for the accepted-token savings to pay off.

The result changed on a narrow Python code-completion prompt:

For this prompt shape, draft 2 was best: 61.57 tok/s, about 35% faster than draft-disabled on the same MTP tag.

I then tested MTP against execution-checked BigCodeBench tasks:

The MTP policy I would use is conservative: enable draft 2 only for predictable function-generation or code-completion prompts, keep it off for prose/planning, and do not use draft 4 broadly until repeated hard-suite checks show stable quality.

Settings that did not help much

• Greedy sampling for speed. It helped repeatability, but did not materially improve throughput.
• Forcing MMQ or cuBLAS. Both landed around 48.3 tok/s, effectively the same operating point.
• NUMA pinning. Binding the server process to one NUMA node measured 48.11 tok/s.
• Two parallel requests. Aggregate generated-token throughput fell in the measured test. This machine is better treated as a single interactive coding host.
• Lower listed 35B CUDA weight quantization. I did not find a smaller listed 35B CUDA Ollama tag such as q3_K_M or q2_K. The practical smaller alternative is a different model, such as a 27B Q4_K_M tag, which would be a separate quality/speed tradeoff.

Research ideas not tested here

I also looked at recent inference work for ideas that may matter if the runtime changes or if these features land in llama.cpp/Ollama later.

• Better speculative decoding. Methods such as Medusa, EAGLE, and newer tree-based speculative methods try to improve acceptance and verification efficiency. They usually need runtime support and model-specific draft heads or draft models.
• Shared-prefix attention. Hydragen targets batches of requests with long shared prefixes. That maps well to coding-agent traffic where several completions share the same repository context.
• More aggressive KV-cache compression. KIVI, TurboQuant, and related work push KV precision below the q8/q4 settings tested here. The q8 result makes this direction interesting, but it depends on practical kernels.
• Serving memory managers. PagedAttention and vAttention matter more for many concurrent requests than for one interactive request.
• Quantization plus kernel co-design. AWQ and QServe are examples. On P40, the useful version would need Pascal-friendly kernels rather than tensor-core assumptions.

The split for this setup is clear enough: prompt-cache discipline works today, q8_0 KV cache works today, shallow MTP works only for predictable completion-shaped prompts, and the more advanced paper ideas need runtime or model changes before they become normal Ollama settings on this machine.

Recommended setup

For this machine, I would run the normal tag with cuda_v12, one loaded model, one parallel request, 8k context for coding, and q8_0 KV cache for long prompts.

Recommended service settings:

[Service]
Environment="OLLAMA_LLM_LIBRARY=cuda_v12"
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_LOAD_TIMEOUT=10m"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_SCHED_SPREAD=1"

For one-off larger-context calls, keep the service default at 4k or 8k and pass the context length per request when needed.

Per-request context override:

{
  "model": "qwen3.6:35b",
  "prompt": "...",
  "stream": false,
  "options": {
    "num_ctx": 16384,
    "num_predict": 512
  }
}

Model unload command:

ollama stop qwen3.6:35b

Takeaways

• The stable short-prompt operating point is about 48 output tok/s.
• For larger coding context windows, q8_0 KV cache with flash attention is the best setting I tested.
• Prompt-cache reuse is the biggest practical latency tool for repeated coding-agent calls.
• MTP is prompt-shape dependent. Draft 2 is useful for predictable code completion and function generation; open-ended prose and planning should avoid it.
• The rapid coding benchmark is useful for quick regressions, and the harder supplement is needed once the easy practical tasks are saturated.