💻Running Locally
Run models on your own hardware · 22 terms
Local Models
7Running a model on your own machine. What the file is, how fast it runs, and what to do when it does not fit.
An AI model that runs on your own machine, not in the cloud.
The file that holds all the numbers a model learned.
A safe file format for model weights. It holds numbers, never code.
How fast a model writes, counted in tokens each second.
If a model is too big for the GPU, part of it runs on the main chip.
How much memory a model needs to run. Weights first, then room for the chat.
AI that runs on the device itself. Think of a phone, a camera or a car.
Model Compression
5Ways to make a model smaller so it fits in less memory and runs faster.
How many bits each weight in a model gets. Fewer bits make a smaller model.
A way to shrink a model. Each weight is kept with fewer bits.
One file that holds a whole model. It is ready to run on your own PC.
The label on a shrunk model file, like Q4_K_M. It says how much it was shrunk.
A way to shrink a model. The parts it needs least are cut out.
Local Runtimes
10The programs that load a model file and run it. Some are for a laptop, and some are for a big server.
The tool that runs a model and gets the answer out of it.
A free engine that runs AI models on most PCs and laptops.
A free tool that gets an AI model and runs it on your PC. It takes one command.
A small text file. It builds your own model in Ollama from a base model.
A free app to find, get and chat with AI models on your own PC.
A free app for your computer. You chat with AI models on it with no internet.
A free tool that serves one AI model to lots of people at once.
Apple's free tool kit. It runs AI models fast on Macs with Apple chips.
A server that takes the same requests as OpenAI's API. Apps can switch to it.
A free chat app in your browser. It talks to AI models you run.