Local AI

Local AI: the explanation

An application loads a model onto your computer and uses its processor, memory and sometimes graphics hardware to generate an answer. The documents you provide can be processed on that same machine. You usually need to download the software and model first. After installation, some tasks can work without an internet connection. Other features, including web searches and calls to external tools, still require network access and may send information beyond your device.

Local processing can be useful when you need to work offline, keep documents within a controlled environment or avoid relying on an external service being available. A community group might sort its archive on its own computer. An organisation might run an assistant on internal servers. The word local mainly describes where the calculations happen. It does not establish who owns the software, what its licence permits or whether every component operates without outside connections.

Results depend on the model, the available hardware and the amount of information you provide. A computer may run out of memory or generate answers slowly. An application described as local might also synchronise conversations, send usage information or switch to a remote model for certain features. You therefore need to inspect its settings and connections. Running locally reduces some data transfers, but you still need updates, access controls and protection for stored documents.

A concrete example

You want to summarise your community group’s meeting notes while travelling without internet access. Before leaving, you install a model that your laptop can run and load the documents into a local application. You check that online search and synchronisation are disabled. The application drafts a summary, which you compare with the original notes to confirm the decisions and the people responsible for them.

Why it matters

Choosing local AI involves balancing control over information against answer quality and hardware requirements. You may also take responsibility for installation and maintenance. Before choosing an application, establish which features run locally and which contact outside services. If it can act as an agent, check the folders it can access and the changes it is permitted to make on your computer.

Common misconceptions

• Local AI does not automatically keep every piece of information on your device, because the surrounding application may include synchronisation, usage reporting or connected tools whose settings and data handling you need to examine separately.
• Local AI is not necessarily unsuitable for demanding work, because a model matched to a specific task may meet your needs, although available memory and processing power constrain which models you can run and how quickly.

Related terms: deploying and hosting

self-hosting

Self-hosting means running software on infrastructure whose administration you manage or control.

LLM observability

LLM observability is the set of methods used to understand an application built with language models through its recorded events and measurements.

MLOps

MLOps is a set of practices for developing, deploying, and maintaining machine learning systems in an organized way.

OpenAI-compatible API

An OpenAI-compatible API is a programming interface that reproduces some or all of an OpenAI API's exchange formats so existing software can connect to it.

Where this site covers it

The term appears on 9 pages of this site. A few of them:

• Local LLM reference guide: software compared, models by machine size, performance, and uses Guide
• Quantization and GGUF: how they work, Q4 and Q8 levels, quality trade-offs, and choosing a file Guide
• Running a local LLM: hardware, software, and limitations Article
• Which GPU do you need to run an LLM? Article
• The cheapest LLM for each use case Article
• LLM benchmarks: how to read model rankings Article
• Ollama step by step: install and run a model locally Article
• Open-source or closed models for your AI agents? Article