Local LLMs: software, hardware, and models

A local LLM runs a language model on your computer or on a machine you administer, using model weights downloaded in advance. The right setup depends on available memory, software, the context you need, and the quality of results on your tasks. Model size alone cannot tell you how useful or fast it will be.

What you actually install

A local installation has several parts: model weights, an inference engine, and an interface. The engine performs the calculations. The interface lets you chat, attach documents, or make the model available to another application.

A single application may package these components together, but they still have different responsibilities. A download problem concerns the files. A loading error may involve the engine or available memory, while a poor answer can also reflect the model, its instructions, and the information it received.

Local describes where the model runs, not every feature of the application. An interface installed on your computer can send requests to a remote provider. Conversely, an application opened in a browser may communicate only with a server on your own network.

Before processing confidential data, identify the selected model and the server address. Also check the extensions, connectors, and services involved in analyzing documents.

This guide is a reference for sizing and operating a local installation. For a shorter discussion of whether this approach fits your needs, see Running an LLM locally: hardware, software, and limitations.

Compare software by the work you need to do

Ollama suits terminal use and integration into applications. LM Studio combines a graphical interface with local server features. llama.cpp provides an engine and tools for more direct control over execution.

Criterion	Ollama	LM Studio	llama.cpp
Starting point	Application and terminal	Graphical interface	Command-line tools
Everyday management	Download and run models by identifier	Select and load models in the interface	Manage files and settings explicitly
Integration	Service accessible to applications	Built-in local server	Project server and tools
Suitable users	People comfortable with a terminal	People who prefer a graphical interface	Developers or administrators
Main point to check	Request destination and selected model	Installed engine and connected features	Compatibility between model and engine

This comparison describes ways of working, rather than ranking generation speed. Two interfaces can use shared underlying technologies while applying different settings.

The official llama.cpp repository describes CPU execution, several forms of hardware acceleration, and the ability to split calculations between the CPU and graphics card. These capabilities were checked for this guide on October 24, 2026.

Start with a single application. Multiple installations make troubleshooting harder and can leave you downloading the same model more than once. The AI tools directory can help you explore other interfaces once you have a clearer idea of your requirements.

Plan memory capacity before choosing a model

Distinguish between system RAM, dedicated graphics memory, and the shared unified memory used by some architectures. Their capacities do not automatically combine into a single pool that is equally fast and straightforward to use.

Memory consumption includes the weights, temporary data used by the engine, and the cache associated with the context. Concurrent requests also affect it. A model file that fits on disk will not necessarily fit in memory during a conversation.

Quantization reduces the precision used to represent some of the model’s numbers. It can lower memory requirements, but you need to assess its effect on quality using your own tasks. The guide to quantization and the GGUF format explains this choice in more detail.

For an initial trial, leave room for your usual applications. A computer that can load a model only after you close everything else may become impractical when you reopen your browser and documents.

The official LM Studio system requirements, accessed on October 24, 2026, recommend at least 16 GB of RAM and specify CPU and operating system requirements. These are software requirements, not a guarantee that any model will run comfortably.

Before buying hardware, record your exact configuration and the amount of memory actually available. Core count, the processor’s product name, and the memory capacity printed on the box do not answer the question on their own.

Models by machine size: cautious starting points

The following table offers editorial starting points as of October 24, 2026, rather than certified minimum configurations. It uses a single model family to make the effect of size easier to compare. It does not present these models as the newest or the best.

Download sizes are those listed in the official Ollama Qwen3 library, accessed on October 24, 2026. They do not represent the total memory required to run each model.

Available machine	First model to consider	Listed download size	Trade-off to check
Laptop with 8 GB of RAM	Qwen3 4B, if enough memory is free	2.5 GB	Short context and limited use of other applications
Computer with 16 GB of RAM	Qwen3 8B	5.2 GB	Responsiveness on the CPU or available acceleration
Machine with 32 GB of RAM	Qwen3 14B	9.3 GB	Quality improvement over the smaller model
Workstation with 64 GB of RAM	Qwen3 32B	20 GB	Acceptable waiting time and hardware cost

If you have a dedicated graphics card, compare the full memory requirement with its available video memory as well. A model loaded partly into system RAM can behave very differently from one loaded entirely onto the card.

With unified memory, the operating system and other applications share the same physical capacity. Do not treat all advertised memory as available for model weights.

These examples do not remove the need to compare other architectures. For a mixture-of-experts model, the number of active parameters alone does not describe all the weights you need to store. Always check the specific file, its format, and whether your engine supports it.

Choose for the task, not just the model size

For rewriting or classifying short texts, start with a modest model. Judge how well it follows instructions: permitted categories, preservation of information, absence of unsupported additions, and an output format you can use.

For document summaries, check coverage of essential information and look for omissions. A fluent answer can leave out a decisive exception. Ask for references to the supplied passages and verify that they match the document.

For programming, use tasks representative of your language and libraries. Plausible code is not enough: compilation, testing, and review are still necessary. A model that can suggest an isolated function may struggle to maintain consistency across a project.

For questions about a document collection, distinguish passage retrieval from answer generation. If the relevant passage never reaches the model, increasing model size does not directly solve the problem.

An agent adds another challenge: selecting a tool, producing its arguments, and interpreting its result. Running locally guarantees none of these capabilities. Begin with read-only operations and retain an approval step for actions that change your data.

Across these uses, evaluate the actual language in your documents, including industry terminology, negations, tables, abbreviations, and proper names. A general benchmark score cannot replace these checks.

Prepare the installation

Before running commands, gather the following:

A computer compatible with your chosen software.
Disk space for the model and additional downloads.
An internet connection to download the software, engine, and weights.
The permissions required to install the software.
A small set of nonconfidential tasks with expected answers.

The walkthrough below uses Ollama. The commands were checked against official sources accessed on October 24, 2026. They do not pin a version, so record the version you actually install if you want to reproduce your tests.

1. Install the software

On Windows or macOS, download the appropriate installer from the official website. On Linux, the official installation documentation provides this command:

curl -fsSL https://ollama.com/install.sh | sh

It downloads and executes a script. In a managed environment, have the script reviewed through your normal process.

The same documentation provides this version check:

ollama -v

Keep the result in your test record. It will help you distinguish a later software change from a change of model.

2. Run a model with an explicit identifier

The official Ollama page for Qwen3 4B provides the following command:

ollama run qwen3:4b

The first run may include a download. Do not confuse this initial wait with the time needed to generate subsequent answers.

Start with a short request whose answer you can verify. Then add a text representative of your work, keeping sensitive data out of this initial testing phase.

Verify execution and prepare for offline use

3. Check which models are loaded

The official Ollama command reference documents how to list installed models and models currently running:

ollama ls

ollama ps

Check that the expected model appears. The processor column helps you identify whether it is running on the CPU, graphics card, or a combination of both.

When you need to free memory, stop the model through the application or use the documented command with its exact identifier. Leaving a conversation does not necessarily mean the weights are immediately unloaded.

4. Verify operation without network access

After downloading everything, temporarily disconnect from the network and repeat a simple request. Also test whether the model loads after you close and reopen the application.

For LM Studio, the offline operation documentation explicitly distinguishes chat, local document processing, and the local server from operations that require internet access, such as finding models or downloading engines.

This test checks offline availability; it is not a complete audit of network communications. An application may retain information and synchronize it later. Examine its storage, network settings, and integrations separately.

Finally, keep a copy of the configuration details you need: model, engine, settings, and data location. Offline operation becomes fragile if reinstalling requires files you have not retained.

Measure performance using comparable conditions

First measure the delay before the first text appears, then generation throughput and total task duration. An answer that starts quickly but continues unnecessarily may be less useful than a concise answer generated more slowly.

Separate the initial loading time from tests performed with the model already in memory. Use the same instructions, documents, context settings, and output limits for every comparison.

Measurement	What it tells you	Common pitfall
Initial loading	Wait before the model is available	Including it in only some tests
Time to first text	Perceived responsiveness	Ignoring the length of the supplied document
Generation throughput	Output production speed	Comparing models that use different tokenization
Total duration	Time until a usable result is ready	Excluding corrections and retries
Verified quality	Usefulness for your task	Judging only the fluency of the writing
Peak memory use	Available operating headroom	Measuring only immediately after loading

Define your criteria before reading the answers: correctly extracted fields, preserved facts, passing tests, and errors that make the result unusable. Record appropriate refusals and answers that acknowledge missing information as well.

Repeat tests and keep track of how much results vary. Temperature settings, competing workloads, and the machine’s thermal state can affect the outcome. This guide gives no universal throughput figure: without a complete configuration and test procedure, a tokens-per-second number is insufficient.

Set the context and understand the real cost

A longer context lets you supply more content, but it increases the work involved and may require more memory. The model’s advertised maximum context is therefore not a recommended setting for every machine.

Start with the documents you actually need. Remove unrelated conversation history and compare the resulting answer before increasing the context. For a large document collection, supplying selected passages may work better than routinely inserting every file.

The official Ollama FAQ, accessed on October 24, 2026, explains context settings and how to interpret memory placement. Use it to check what your installation is actually running.

Local costs include hardware, electricity, storage, and administration time. They also include unusable answers and the human work needed to correct them.

Compare completed tasks at a similar level of quality. A computer you already own and use occasionally requires a different budget from a dedicated server running continuously.

If you are considering a purchase, first test your workload on a configuration you can access. Check that the resulting quality justifies the investment. More memory may let you load a different model without delivering enough improvement for your needs.

Common problems

The model will not load. Check the download, engine compatibility, and free memory. Reduce the context, close memory-intensive applications, and then try a smaller variant.

Generation is slow despite having a graphics card. Check where the calculations are actually running and whether the hardware is supported. Then compare performance with a smaller model, rather than changing every setting at once.

Memory use grows during conversations. Examine conversation length and concurrent requests. An installation that works reliably for one person may run out of memory when several sessions use the server.

Answers are inconsistent or poorly formatted. Check the exact model, engine, and expected conversation format. Return to a minimal request before adding documents and constraints.

Documents seem to be ignored. Check text extraction and the passages actually sent to the model. A scanned file may need optical character recognition, while a table may lose its structure during extraction.

The server works on its own machine but cannot be reached elsewhere. Distinguish local listening settings, network access, and firewall rules. Do not simply expose the service to the internet to work around the problem.

After an update, rerun your task set before replacing the installation you use every day. Retain the files needed to restore the previous configuration.

Checklist before regular use

A successful installation produces verifiable answers within an acceptable time, with enough memory headroom and a known destination for your data. Validate these points before increasing model size or adding users.

Frequently asked questions

Can I run a local LLM without a graphics card?

Yes, provided the inference engine supports your computer's CPU. Available memory and response time then become the main selection criteria. Start with a small model before considering a hardware purchase.

Does a local LLM work without an internet connection?

Inference can work offline once the software, its engine, and the required model weights are installed. Web searches, remote providers, and some connected tools still need network access.

Does a local model automatically learn from my documents?

No. Providing a document in a conversation generally does not change the model's weights. An application may keep the document or build a search index, creating a separate store of data that you need to manage.

Can a local LLM replace an API for a business?

That depends on workload, quality requirements, and operational constraints. An API removes the need to manage hardware. A local installation gives you more control over execution but requires maintenance and enough capacity.

Can every downloadable model be used commercially?

No. Being able to download model weights does not establish your right to use them. Check the license for the exact model, any additional terms, and the license of the software used to run it.

Local LLM reference guide: software compared, models by machine size, performance, and uses