How to Run LLMs Locally on a Jetson Orin Nano with Ollama

Slug: /run-llms-locally-on-a-jetson-orin-nano-with-ollama Meta description: Run LLMs locally on a Jetson Orin Nano with Ollama: native vs Docker install, Open WebUI, and memory-budget guidance for 4GB and 8GB boards.

For engineers and students prototyping edge-AI products, running real language models on a small, affordable board is now a matter of knowing what fits, not whether it can be done. Few devices make that as approachable as NVIDIA’s Jetson Orin Nano, and the fastest way to run LLMs locally on a Jetson Orin Nano with Ollama exploits Ollama’s first-class, CUDA-enabled support for the Jetson platform. According to the official NVIDIA Jetson AI Lab tutorial (source repo), Ollama installs with a single command and works out of the box on recent JetPack releases. This walkthrough covers the two supported installation paths (native and Docker), wiring up a browser interface with Open WebUI, and an honest memory-budget table so you know which quantized models fit on a 4GB versus an 8GB board before downloading gigabytes of weights.

Why Run LLMs Locally on a Jetson Orin Nano with Ollama

Ollama is a popular open-source tool that runs LLMs locally as a small HTTP server plus CLI. Its appeal on Jetson is the removal of friction: the official installer ships CUDA support for the platform, and the Jetson AI Lab documents support across the Orin family — down to the Orin Nano 8GB — on JetPack 5 and 6. Two practical notes from the tutorial matter before anything else. First, storage: an NVMe SSD is highly recommended, both for speed and because the Docker image alone is about 7GB and models run past 5GB. Second, performance expectations: Ollama uses llama.cpp underneath, and NVIDIA reports it reaches “roughly half of peak performance versus faster APIs like NanoLLM” — generally “fast enough for text chat.” The realistic promise of this stack is a responsive local chatbot and a simple OpenAI-compatible endpoint, not a high-throughput production server.

Option 1: Native Installation

The native path is the one-command route. On a Nano with JetPack 5 or 6, run the official installer:

curl -fsSL https://ollama.com/install.sh | sh

The installer creates a system service that starts ollama serve on boot, so the ollama command is immediately usable. Run any library model:

ollama run gpt-oss:20b

If a model is too large for the board’s memory, the Jetson AI Lab guidance is explicit: drop to a smaller model from the Ollama library.

Option 2: Docker Container

Docker lets you run Ollama without modifying the host system. For Orin boards on JetPack 6, pull the container and run it with NVIDIA runtime support:

docker pull dustynv/ollama:r36.2.0

# models cached under the jetson-containers data directory
jetson-containers run --name ollama $(autotag ollama)

# or models cached under your home directory
docker run --runtime nvidia -it --rm --network host -v ~/ollama:/ollama \
  -e OLLAMA_MODELS=/ollama dustynv/ollama:r36.2.0

Once inside, the container reports its OLLAMA_HOST, OLLAMA_LOGS, and OLLAMA_MODELS paths; you then issue the same ollama run commands as native. The client itself can live inside or outside the container once the server is up — handy when your application code lives on the host.

Native vs Docker: choose native for the simplest, most maintenance-free setup; choose Docker when you want the model stack isolated from your system. Both end at the same Ollama API on port 11434.

Add a Web Interface with Open WebUI

A CLI is fine for testing, but most users want a browser chat UI. The Jetson AI Lab shows an Open WebUI container pointed at the local Ollama server:

docker run -d --network=host -v open-webui:/app/backend/data \
  -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui \
  --restart always ghcr.io/open-webui/open-webui:main

Open your browser to http://JETSON_IP:8080; the tutorial notes you can create a local (fake) account whose credentials live only on the device. Open WebUI fronts any pulled Ollama model and exposes the same REST interface to your own scripts.

The Memory Budget: Which Quantized Models Fit on 4GB vs 8GB

The Orin Nano Developer Kit ships in 4GB and 8GB unified-memory variants, and that shared memory — not peak FLOPs — decides what you can run, because weights, the KV cache for context, and the OS all compete for it. Quantized GGUF models (typically Q4_K_M) shrink weights dramatically, but the fit is qualitative: real headroom depends on quant level and the context length you allow. The table below gives typical 4-bit file sizes for common Ollama library models and what they realistically mean on each board.

Sizes are approximate 4-bit (Q4_K_M) figures from the Ollama library and vary a little with quant and architecture; confirm the exact file with ollama show <model>, and remember the KV cache consumes headroom on top of the weights. The rule of thumb: ~2GB of weights fits a 4GB Nano with a small context window; anything above ~3GB is an 8GB-board job.

The Honest Close: What a 4GB Board Can and Cannot Do

The same AI that is easy to oversell is easy to oversave on, so let us be precise. A 4GB Orin Nano can run small models — roughly 1–3 billion parameters at 4-bit — comfortably, which is enough for instruction-following, summarization, classification, and lightweight chat with short contexts. What it cannot do is run 4B+ models with meaningful context, hold an 8B model at all, or carry vision-bundled variants whose encoders spill into CPU memory and silently halve performance — a lesson documented in first-party benchmarks on the 8GB Nano, where packaging, not the engine, hid the biggest losses, and Ollama and llama.cpp ran within a few percent on identical fully-offloaded files.

On model compression specifically, honesty requires citing the recent survey “Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment”. Its central finding is that no single compression technique wins across tasks and hardware: for question answering, quantization outperformed structured pruning at equal precision, while for segmentation the ranking reversed and pruning shrank models by ~80% at near-constant accuracy. Worse, a lossy technique can even inflate the deployed artifact or raise latency by breaking the quantization layout. As the survey concludes, “the right technique depends on the task, the model, and the hardware.” For a Nano user, the translation is: before blaming the board for a slow or degraded model, question the quantization scheme and the context you allowed, because the failure is frequently in the trade-off, not the silicon. A 4GB Nano is a genuinely useful edge-LLM device when matched with the right small, well-quantized model — and candidly limited once you demand larger models, long contexts, or vision. Plan the memory budget first, and the board will deliver.

Reviewed and approved by Prof. Ajay S., professor at a State University in Delhi.