Local LLM: hardware, setup, and limitations

To run a local LLM on your computer, install software that can run models, download a compatible model, and load it into memory. You can then chat without sending your prompts to a remote service, provided you select local execution and check any connected features. The main limitations are available memory, response speed, and how well the model handles your tasks.

What actually runs on your computer

A local setup has several parts: a chat interface, an inference engine that performs the calculations, and the model files, known as weights. An application installed on your computer can also act as an interface to a remote model. Having an app on your desktop therefore tells you little about where the processing happens.

LM Studio provides an interface for downloading, loading, and chatting with models. Ollama also lets you run models locally and make them available to other applications. Its getting-started workflow explicitly distinguishes choosing a local model from using cloud models, according to the official documentation accessed on October 24, 2026.

For your first attempt, choose a narrow task: rewording an email, summarizing a document that contains no sensitive information, or organizing notes. Define a satisfactory result before installing anything. This gives you a way to judge whether the system is useful, beyond how smoothly the conversation flows.

Running locally can be a good fit when you want to work offline, control the files being used, or experiment without relying on a remote service. It also means managing downloads, disk space, updates, and backups. The local LLM reference guide covers architectures and use cases beyond this initial setup.

Hardware: start with available memory

Before downloading a model, check your operating system, processor, installed RAM, and any graphics card, including its dedicated memory. Check available disk space as well. You can find these details in your system settings and resource-monitoring tools.

RAM is the system memory that holds, among other things, data used by running programs. VRAM is the graphics card’s dedicated memory. In a unified-memory architecture, components share the same pool of memory, so you still need to leave enough room for the operating system and other applications.

Requirements vary by application. As a dated reference point, LM Studio recommends at least 16 GB of RAM on Mac and Windows, plus 4 GB of dedicated VRAM on Windows. Its documentation also mentions running small models on a Mac with 8 GB. These are software recommendations, not a guarantee that every model will work, according to the official system requirements accessed on October 24, 2026.

Setup	What to check first	Suggested first test	Limitation to watch
Computer without a dedicated GPU	CPU compatibility and available RAM	Compact model with a short text	Response time
PC with a dedicated graphics card	GPU compatibility and available VRAM	GPU-accelerated loading, if supported	Spillover into system RAM
Computer with unified memory	Memory available across the whole system	Model that leaves room for other applications	Memory pressure
Computer used for several purposes	Resources available during your normal work	Test with your usual applications open	Overall slowdown

Do not choose a graphics card based on its product name alone. Memory capacity, compatibility with the inference engine, and measured performance on your intended workload should guide the purchase. The article Which GPU do you need to run an LLM? examines that decision in more detail.

Model size: the file is only part of the memory requirement

Download size is a useful first filter, but it does not tell you how much memory a conversation will use. The model also needs memory for calculations and for information associated with the context. A setup that starts successfully may therefore reach its limits when you add a long document.

Quantization reduces the precision used to represent the weights. This makes them smaller, with a possible loss of quality. The quantization tool’s documentation explains this tradeoff and distinguishes several formats, including Q4 and Q8, in its technical overview accessed on October 24, 2026.

Do not assume that a smaller file will retain exactly the same capabilities. Test for the errors that matter to you: incorrect names or numbers, failure to follow a required format, poor French-language performance, or an inaccurate summary. A polished answer can still leave out something essential.

Another common mistake is immediately setting the context to the maximum advertised length. Start with a short conversation, then gradually increase the amount of text. The useful setting is one that accommodates your task while leaving enough memory available.

Keep a record of the exact file you downloaded and the settings you used. Without that information, you may compare different variants while believing you are testing the same model. For more on formats and their tradeoffs, see the guide to quantization and GGUF.

Software: choose a workflow before adding more layers

An interface that combines downloading and chatting reduces the number of components you need to configure when starting out. You can see which model is loaded and begin with a simple exchange. Integration with an editor or business application can come after you have confirmed that this basic setup works.

Users who want more direct control over the engine can look at llama.cpp. Its repository documents CPU execution, several forms of GPU acceleration, and ways to split calculations between the CPU and GPU. The available options depend on your hardware and how the software was compiled, according to the official repository accessed on October 24, 2026.

Avoid starting with a web interface, an inference engine, a document manager, and a collection of extensions all stacked together. Each layer adds settings, logs, and potential network connections. If responses become slow or requests go to a remote service, those extra components make the cause harder to identify.

To choose between the two applications introduced above, see the LM Studio vs. Ollama comparison. The goal here is to build a local setup whose behavior you can check.

Before choosing software, pay particular attention to:

Compatibility with your operating system and hardware.
The model formats it supports.
Whether you can identify the model actually being used.
Network settings and connected features.
How to delete conversations and downloaded files.

First run: a setup without the terminal

Prerequisites: a compatible computer, an internet connection for downloads, available disk space, and a sample text containing no sensitive information. The following workflow follows the official LM Studio getting-started guide, accessed on October 24, 2026.

Download the application from its official website. Choose the installer for your operating system. Check the requirements before installing, especially on an older computer.
Open the model discovery area. Select a model designed for conversation and compatible with the available engine. Read its model card, license, and file size before starting the download.
Load the model in the chat area. Select the downloaded file in the model loader. Use conservative settings for your first test, particularly for context length.
Send an instruction whose result is easy to check. For example, ask the model to reword a paragraph without changing any names, dates, or amounts. Compare the output with the original.
Then try your intended task. Use a representative document, still without sensitive data. Record errors and response times before changing the settings.
Check that it works offline. Once you have received an answer, temporarily disconnect from the network and start a new conversation using the model you have already downloaded.

This last test establishes that your particular workflow can run without a network connection at that moment. It does not audit every exchange that took place earlier or might take place after you reconnect.

For a workflow that makes more use of the terminal and engine diagnostics, the step-by-step Ollama tutorial complements this graphical setup.

Privacy: check the entire processing chain

The main privacy benefit of running locally is the ability to keep processing on your computer. That needs to hold throughout the workflow: the model itself, document reading, retrieval of relevant excerpts, and any tools called along the way. A single remote service can change where some of your data goes.

LM Studio states that conversations with downloaded models, local document processing, and its local server can work offline. Searching for and downloading models, engines, or updates requires network access, according to its documentation on offline operation.

Ollama also has a setting to disable its cloud features. Its FAQ says this disables cloud models and web search. It also explains how to identify whether a model is loaded on the CPU or GPU, in the official documentation accessed on October 24, 2026.

If you plan to work with confidential documents, check system backups, synced folders, and extensions separately. A conversation can remain local while it is being generated, then be copied elsewhere by an automatic backup.

Keep server access limited to what your workflow requires, too. If you enable access over the network, check who can reach it and how access is protected. When connecting tools, grant only the read and write permissions they need: local execution does not remove the risk of an incorrect action.

Measure quality, speed, and the actual cost

Evaluate the model on your own tasks before considering a hardware purchase. Prepare prompts whose answers you already know and documents whose summaries you can verify. Include incomplete examples to see whether the system recognizes when information is missing.

Assess quality and speed separately. A fast but incorrect answer does not solve your problem, while a correct answer may still be impractical if the wait disrupts your work. Watch loading time, the delay before the answer begins, and the total time needed to complete the task.

The language model profiles can help you document your shortlist. The benchmark lab helps put public evaluations in context, but you still need to compare those findings with your own use.

For budgeting, include any hardware you buy, electricity, storage, and time spent on maintenance. If you already own the computer, distinguish that past expense from the additional cost of this project. For occasional use, buying equipment solely to avoid remote-service charges requires a cost comparison first.

Running locally does not automatically give the model access to recent information, either. Downloading a model does not provide it with ongoing web search. Ask for sources when the task calls for them, and verify the documents it uses, even when its answers sound confident.

Common problems

The model will not load. First check compatibility between the file format and the engine, then check available memory. Close demanding applications, reduce the context length, or try a smaller file. Free disk space is no substitute for the memory needed to run the model.

Generation is too slow. Check whether GPU acceleration is actually being used and whether the system is swapping large amounts of data to disk. Test a short conversation before changing hardware. Adjust only one setting at a time so you can understand its effect.

Answers are inconsistent or ignore instructions. Start a fresh conversation with a simple request. Check that you selected a variant intended for conversation and that its format is supported. Then compare another variant on the same task.

Chat works, but documents do not. Document processing introduces its own components and settings. Check which components are installed and where their calculations take place. Start with a short text document before adding more complex files.

The application unexpectedly requests a connection. Identify the feature involved: a download, an update, a remote model, or a connected tool. Stop using sensitive data until you understand where the data is going.

Checklist before regular use

The software and model come from identified sources, with a license suitable for my use.
The computer has enough memory left for my other applications.
Quality and response times are suitable for representative tasks.
The intended workflow works offline if that is a requirement.
Network connections, backups, and file access are under control.

Frequently asked questions

Can I use local AI without an internet connection?

Yes, provided you have downloaded the model and the components needed to run it. Have any documents you want to use ready before disconnecting, too. Features that contact a remote service will remain unavailable.

Do I need a graphics card to run a local LLM?

Some inference engines can run models using your CPU and system memory. Compatibility depends on the software and model you choose. Before buying a graphics card, check whether your current computer is fast enough for your tasks.

Does local AI automatically learn from my documents?

Adding a document to a conversation does not automatically retrain the model. The application may pass relevant excerpts to it when generating an answer. Whether the document, its index, and the conversation are retained depends on the software settings.

Is running a local LLM free?

Running a model locally can avoid per-request charges from a remote provider. It still uses your computer, electricity, and time spent on maintenance. Check the software terms and the model license separately.

Running a local LLM: hardware, software, and limitations