How to run AI locally on your own computer: a step-by-step tutorial

Running AI locally means downloading an open-weights model and executing it on your own computer, without sending your text to an online service. With a tool such as Ollama or llama.cpp, setup takes a handful of commands; the real decision is how large a model your machine can handle.

What “running AI locally” actually means

A large language model is a big file of numbers called weights. When the developer publishes them, it is an open-weights model: you can download the weights and do the computation yourself. That computation, which produces the answer, is called inference.

There are three good reasons to do it at home:

• privacy: your documents never leave the machine;
• cost: no per-token bill, only hardware and electricity;
• independence: the model works offline and does not change unless you change it.

The trade-off is real: less capability than the best online models, plus some setup work. This tutorial aims at a concrete first result: a model that answers in your terminal, and then answers your own programs.

Prerequisites: what your machine needs

Before installing anything, check three things.

1. The operating system. According to the Ollama documentation checked on October 22, 2026, you need Windows 10 22H2 or newer, or macOS Sonoma (version 14) or newer. On a Mac, Apple M-series chips use both CPU and GPU; Intel Macs run on the CPU only. Linux is supported too.

2. Memory. This is the deciding factor. For good speed, the model must fit in VRAM, the memory on your graphics card (GPU). Otherwise it can run from system RAM, more slowly. A model’s size depends on its number of parameters and on its quantization, meaning the precision used to store each weight. Files in the GGUF format, which both tools use, are most often quantized to around 4 bits.

3. Disk space. The Ollama Windows documentation asks for at least 4 GB for the software itself, plus room for models.

Here are some orders of magnitude from the official Ollama library, checked on October 22, 2026 (download size, depending on the variant):

Model (Ollama tag)	Listed size	Suitable machine
qwen3.5:4b	3.3 to 4.0 GB	Recent laptop, even without a dedicated GPU
qwen3.5:9b	6.6 to 7.6 GB	Graphics card or Mac with comfortable memory
gemma4:e4b	6.6 to 9.5 GB	Same
gemma4:26b	16 to 19 GB	Well-equipped machine
qwen3.5:27b	17 to 20 GB	Well-equipped machine
gemma4:31b	19 to 20 GB	Well-equipped machine

The right-hand column is our own reading, not an official requirement. Always leave headroom beyond the file size: the model needs working memory for the conversation context.

Picking an open-weights model

Developers release more and more open models, in every size. The table below lists a few that suit local use, with the site’s data:

Model	Developer	Release	Context (tokens)	Input ($/M tokens)	Output ($/M tokens)	Open weights
Gemma 4 31B IT	Google DeepMind	April 2, 2026	262 k	$0.14	$0.40	Yes
Gemma 4 26B A4B IT	Google DeepMind	April 2, 2026	262 k	$0.10	$0.34	Yes
Qwen3.6 27B	Alibaba Cloud (Qwen)	April 22, 2026	262 k	$0.33	$2.7	Yes
Qwen3.6 35B-A3B	Alibaba Cloud (Qwen)	April 17, 2026	262 k	$0.17	$1	Yes
Ministral 3 8B 2512	Mistral AI	December 2, 2025	262 k	$0.15	$0.15	Yes
Granite 4.2 8B	IBM (Granite)	August 25, 2026	131 k	$0.06	$0.25	Yes

The prices shown are those of hosting providers that serve these models through an API: locally, you do not pay per token. Two families come up again and again in consumer tools: Gemma, from Google DeepMind, whose largest version is described on the Gemma 4 31B IT page, and Qwen, from Alibaba. Models labeled “A4B” or “A3B” are mixture-of-experts models: only part of the weights works on each word, which speeds things up, but all the weights still have to fit in memory.

To put their level in context, here is the Epoch AI ranking index as of October 22, 2026, which combines several demanding tests, with a leading closed model as a reference point:

Capability index (out of 100)

• Claude Opus 5.595.9
• Kimi K2.684.4
• Qwen3.6 27B72.2
• Qwen3.6 35B-A3B55.3
• Qwen3.5 9B49.6

Kimi K2.6 has open weights, but its size limits it to servers. The models that fit on a personal computer score clearly lower. The full list, with licenses, is on our open-source models page.

Step 1: install Ollama

Ollama is the easiest route: a program that downloads models, keeps them in memory and serves them to your applications. The commands below come from the official README.

On macOS or Linux, in a terminal:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, in PowerShell:

irm https://ollama.com/install.ps1 | iex

If you would rather not run a downloaded script, the same README offers a regular installer for Mac and Windows, as well as an official Docker image. Get into the habit of reading a script before running it: it is basic security hygiene.

Step 2: download and run your first model

A single command downloads the model if needed, then opens a chat in the terminal:

ollama run gemma4

Without a tag, Ollama picks the default variant, here gemma4:e4b according to its library. On a more modest machine, choose a smaller tag, for example ollama run qwen3.5:4b. Type your question, then /bye to quit.

Other useful commands, documented in the CLI reference:

ollama pull qwen3.5:9b   # download without running
ollama ls                # list installed models
ollama ps                # show models loaded in memory
ollama stop gemma4       # unload a model
ollama rm gemma4         # delete a model from disk

Expected result: the model starts answering within seconds, word by word. Turn off Wi-Fi and ask another question: the answer still comes. That is your proof that everything happens on your machine.

Step 3: call the model from a program

Ollama also works as an inference server: according to its documentation, its API listens on http://localhost:11434, on your computer only. Try it with curl, adapting the README example:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{"role": "user", "content": "Explain the GDPR in three sentences."}],
  "stream": false
}'

Even handier, Ollama offers an OpenAI-compatible endpoint. According to the compatibility page, the client library requires a key, but Ollama ignores it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
response = client.chat.completions.create(
    model="gemma4",
    messages=[{"role": "user", "content": "Suggest three titles for an article about composting."}],
)
print(response.choices[0].message.content)

This detail matters: most tools written for an online API accept a custom base URL. The same page says tool calling is supported, so you can plug a local model into an agent framework. Atomic Agents, for instance, lists Ollama among its providers. For the next steps, our guide on how to build an AI agent, from no-code to code covers the full process, and our AI agent frameworks comparison helps you pick a tool.

Alternative: llama.cpp, for full control

llama.cpp is the engine Ollama builds on. Using it directly gives you more knobs: how many layers go to the GPU, context size, the exact GGUF file. According to its README checked on October 22, 2026, it supports Apple chips (Metal), NVIDIA (CUDA), AMD (HIP) and Intel (SYCL) GPUs, Vulkan and plain CPUs, with quantization from 1.5 to 8 bits.

Installation, per the README:

# macOS or Linux
curl -LsSf https://llama.app/install.sh | sh

# Windows (PowerShell)
irm https://llama.app/install.ps1 | iex

Prebuilt binaries, a Docker image and building from source are also available. Then two commands:

# download a model from Hugging Face and chat with it
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# start an OpenAI-compatible server with a web interface
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

The example uses a very small model, ideal for checking that everything works. According to the server documentation, the -hf option defaults to Q4_K_M quantization, and the server listens on 127.0.0.1, port 9931: open http://localhost:9931 in your browser to chat, or point your programs at http://localhost:9931/v1.

Criterion	Ollama	llama.cpp
Getting started	Very easy, one command per model	More technical, many options
Catalog	ollama.com library, ready-made tags	Any GGUF file, notably on Hugging Face
API	Native and OpenAI-compatible, port 11434	OpenAI-compatible, port 9931 by default
Interface	Terminal and desktop app	Terminal and built-in web UI
Best for	Starting fast, wiring up an app	Tuning a machine, testing variants

Check yourself Your laptop has 8 GB of memory in total and no dedicated graphics card. Which model do you try first?

A small one, such as qwen3.5:4b (3.3 to 4.0 GB in the Ollama library) or the llama.cpp demo model. A 7 GB model would fill almost all the memory, operating system included: it would start badly or run very slowly.

Troubleshooting the most common problems

• It is painfully slow. The model probably does not fit in GPU memory and spills over to the CPU. ollama ps shows where it is loaded. Switch to a smaller tag or a stronger quantization.
• “Out of memory” or a crash at load time. Same cause, or a context window set too large: conversation memory comes on top of the weights.
• Your disk fills up. Remove unused models with ollama rm. On Windows, the OLLAMA_MODELS variable lets you store them on another drive, according to the documentation.
• Weak answers in your language. Try a larger model or another family: small models are often most at ease in English, less so in other languages.

Going further: your documents and security

A local model becomes truly useful when it works on your own files. That is the idea behind RAG, which gives your documents to an AI: you retrieve the relevant passages, then hand them to the model. Ollama exposes an embeddings endpoint for this, according to its compatibility page.

On the security side, keep the server on localhost. If you open it to your network, any connected device can use it. And a local model is not immune to prompt injection: as soon as it reads outside content and has tools, apply the same rules as for an online agent, covered in our guide on AI agent security.