Setting Up a Local AI Server in 2026: GPU Selection and Ollama

In this tutorial, we will guide you step-by-step through setting up a local AI server in 2026. You’ll learn to choose the right GPU for your project, install necessary software, and configure Ollama to run AI models locally with Docker. This guide covers everything you need to create an autonomous and controlled development environment.

Prerequisites

OS: Ubuntu 22.04 LTS
CPU: Minimum Intel Core i7 or AMD Ryzen 5
GPU: NVIDIA RTX 3060 minimum (see the section on choosing a GPU)
RAM: 16 GB minimum
Disk Space: 50 GB for operating systems and Docker images
Open Ports: 80 (HTTP), 443 (HTTPS) if needed

Step-by-step installation

Step 1: Choose the Right GPU

Select a GPU that fits your budget and meets the needs of your AI models. Here are some recommended options:

NVIDIA GeForce RTX 4090 24 Go : For maximum performance.
Intel Arc A770 16 Go : An economical alternative option. Note: it does not use the NVIDIA Container Toolkit. Ollama uses it through the Vulkan backend of the ollama/ollama image, provided the container is given access to /dev/dri instead of the NVIDIA deploy section (steps 4 and 5 are NVIDIA-specific).

Step 2: Install Docker and Necessary Dependencies

Download and install Docker on your server. Follow the official documentation to install Docker CE (Community Edition) :

curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
sudo usermod -aG docker $USER
sudo systemctl enable --now docker.service

Step 3: Prepare the Operating System

Install necessary packages for Docker and configure the system for optimal performance:

sudo apt update && sudo apt install -y curl git

Step 4: Configure Ollama

Create a dedicated folder (mkdir -p ~/ollama && cd ~/ollama), then create a docker-compose.yml file in it to configure your AI server (step 6 is run from this folder):

services:
  ollama:
    restart: always
    image: ollama/ollama
    container_name: ollama_topo
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
    volumes:
      - ollama:/root/.ollama
    ports:
      - "11434:11434"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
volumes:
  ollama:

Step 5: Configure GPU with NVIDIA-Docker

Install the NVIDIA drivers and configure Docker to use your GPU (reboot the server after installing the drivers and check with nvidia-smi before installing the NVIDIA Container Toolkit):

sudo ubuntu-drivers autoinstall
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 6: Launch Ollama Server

Run the docker-compose.yml file to start your Ollama server:

docker compose up -d

Verification checks

Verify that your server is running with the following commands:

docker ps | grep ollama
curl http://localhost:11434

You should see a Docker container named ollama_topo, and curl should reply “Ollama is running”.

Security and best practices

To secure your local AI server, install HTTPS with Caddy or Nginx. Use strong passwords for all SSH connections and configure two-factor authentication (2FA) if possible.

Keep your operating systems and software up to date regularly to maintain maximum security levels.

Troubleshooting common errors

Symptom : The server does not start. Cause: NVIDIA drivers are either not installed or incorrectly configured. Solution: Reinstall the NVIDIA drivers and ensure Docker is set up to use the GPU.
Symptom : Error 404 when accessing the server via HTTP/HTTPS. Cause: Incorrect port configuration or network issue. Solution: Check the docker-compose.yml file configuration and make sure port 11434 is allowed in the firewall for your local network only (the Ollama API has no authentication: never expose it directly to the Internet).

FAQ

The question? Which GPU should I choose between RTX 4090 and Arc A770?

The RTX 4090 is more powerful but expensive, while the Arc A770 offers good value for money.

The question? Does Ollama support many different AI models?

Yes, Ollama supports a wide variety of AI models, including those specific to text and visual content generation.

Host this yourself

No hardware to manage? This kind of service runs on a managed VPS, ready in minutes. See PlanetHoster plans