Practical guide

Why Is Ollama Using the CPU Instead of My GPU?

Start with ollama ps while a model is loaded. A CPU/GPU split means Ollama is already using the GPU. If a small model stays at 100% CPU, check whether the server can see the device.

By InfoIsland Editorial · · Documentation reviewed

A processor and a graphics card connected by parallel blue paths.
Original AI-generated editorial illustration; not a product screenshot.

Read the processor column first

Run ollama run YOUR_MODEL, replacing YOUR_MODEL with a name from ollama list. Send one short prompt and leave the session open. In a second terminal, run ollama ps. Look for the row with the same model name.

  • 100% GPU. Ollama loaded the model onto the GPU. If replies still feel slow, compare a short prompt with your normal workload. A long prompt can take time to process before the first generated token.
  • A CPU/GPU split. The model is using both system memory and GPU memory. A smaller context or a smaller set of weights may let more of the model fit on the GPU.
  • 100% CPU. Check with a small model that comfortably fits your hardware. If that model stays on CPU too, look at device visibility and server logs.

The percentages describe model placement. For live utilization and available device memory, use your GPU’s monitoring tool. If ollama ps is empty, send another prompt and check again while the model is resident.

Compare two context sizes with the same model

In the interactive ollama run session, enter the setting below and then send a short prompt. Record PROCESSOR and, on versions that show it, CONTEXT from the other terminal.

/set parameter num_ctx 2048

Repeat in a fresh session with 8192 if the model supports that context and your hardware can handle it. Keep the model, prompt, and other loaded workloads the same. Close the first session and use ollama stop YOUR_MODEL before starting the next run if you need to unload the first configuration. If the active context doesn’t change, check whether your app or API client is supplying its own value.

Suppose the first run shows 100% GPU and the second shows a split. That points to the larger memory budget at the higher context setting. Keep the lower setting if it fits the prompts you actually use, or choose smaller weights to make room for longer prompts. The KV-cache calculator helps you compare those context sizes before another run. Results depend on the model and runtime settings.

When a small model still uses only the CPU

  1. Check the exact GPU and driver against the hardware support list. On an NVIDIA machine, nvidia-smi gives you a quick check for driver visibility and occupied memory.
  2. Check the machine that runs the Ollama server. A local terminal may be talking to a server in a container, virtual machine, or on a different host. The device must be visible where inference runs.
  3. Read the startup and model-load logs. On a Windows desktop installation, open %LOCALAPPDATA%\Ollama in Explorer and look for server.log. Check the timestamp around your test for device discovery, available memory, or backend errors.
  4. If a driver or device-visibility setting changed, restart Ollama and repeat the small-model test before trying the larger model again.

What to change next

If the small model runs fully on the GPU and the larger one splits, use the GPU model-fit finder and compare candidates at your real context size. If both stay on CPU and the logs show no usable device, resolve GPU detection first. If the error comes from llama-server and names a CUDA allocation, follow the llama.cpp memory troubleshooting guide.

The processor outcomes above are examples to help interpret your own results. Use your installed version’s output when reporting a diagnosis.

Sources and further reading