Run a Local LLM on Your Server with Ollama and Open WebUI: CPU Sizing, Install, HTTPS and API

You can run a private ChatGPT-style assistant on your own server without a GPU. Ollama downloads and runs open-weight language models such as Qwen, Llama and Gemma behind a simple API, and Open WebUI adds a chat interface with users, history and document upload. Because everything runs on your server, prompts and documents never leave it. This guide covers choosing a server size, installing Ollama, picking a model that runs well on CPU, adding Open WebUI with HTTPS and using the API from your own applications.

Step 1: Choose the Right Server Size

On CPU, the amount of RAM decides which models you can load, and the number and speed of the cores decide how fast answers appear. Quantized models (the default Q4 builds in Ollama) need roughly the following memory:

Expect a few words per second for 7–8B models on a modern multi-core CPU: fine for a small team or background jobs, too slow for hundreds of simultaneous users. For heavier use, choose a dedicated server with more cores and RAM rather than a small VPS, and keep other memory-hungry services off the same machine.

Step 2: Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

systemctl status ollama --no-pager
ollama --version
ss -lntp | grep 11434        # should show 127.0.0.1:11434

The installer creates an ollama system user and a systemd service that listens on 127.0.0.1:11434. Models are stored in /usr/share/ollama/.ollama/models; make sure that disk has room (a 7B model is about 4–5 GB).

Step 3: Download and Test a Model

# Strong multilingual model, good at Chinese and English
ollama pull qwen2.5:7b

# Smaller and faster alternative
ollama pull llama3.2:3b

ollama run qwen2.5:7b "Explain in three sentences what a reverse proxy is."
ollama list        # downloaded models
ollama ps          # loaded models and how much memory they use

Browse the model library on ollama.com for other models and sizes. Start with a 3B or 7B model, check the speed, and only move up if the answers are not good enough.

Step 4: Install Open WebUI

Open WebUI runs in Docker (see our Docker installation guide). The container must reach Ollama on the host, so let Ollama listen on all interfaces and make sure the firewall blocks port 11434 from outside.

# Let Ollama accept connections from Docker containers
mkdir -p /etc/systemd/system/ollama.service.d
cat > /etc/systemd/system/ollama.service.d/override.conf <<'EOF2'
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
EOF2
systemctl daemon-reload && systemctl restart ollama

# Allow Docker networks to reach Ollama, block everyone else (ufw: rules are matched in order)
ufw allow from 172.16.0.0/12 to any port 11434 proto tcp
ufw deny 11434/tcp
# firewalld: simply do not open 11434; Docker's own zone already lets containers reach the host

# Start Open WebUI, reachable only from the server itself until Nginx is in front
docker run -d --name open-webui --restart always \
  -p 127.0.0.1:3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

From another machine, confirm that http://YOUR_SERVER_IP:11434 does not answer. If it does, fix the firewall before continuing.

Step 5: Publish Open WebUI over HTTPS

Point a subdomain such as ai.example.com to the server, get a certificate (see free SSL with Certbot or acme.sh) and proxy to the container. Open WebUI streams answers over WebSockets, so keep the upgrade headers and a long read timeout.

server {
    listen 443 ssl;
    http2 on;
    server_name ai.example.com;

    ssl_certificate     /etc/letsencrypt/live/ai.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ai.example.com/privkey.pem;
    client_max_body_size 50m;      # document uploads

    location / {
        proxy_pass         http://127.0.0.1:3000;
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection "upgrade";
        proxy_set_header   Host $host;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_read_timeout 600s;
    }
}

Open the site and create the first account immediately; it becomes the administrator. Then, in the admin settings, turn off new sign-ups or require admin approval, and create accounts for your team yourself.

Step 6: Use the API from Your Applications

Ollama offers its own API and an OpenAI-compatible endpoint, so many tools and SDKs work by just changing the base URL. Call it from the same server, or from your other servers over a private network or an SSH tunnel.

# Native API
curl http://127.0.0.1:11434/api/generate -d '{
  "model": "qwen2.5:7b",
  "prompt": "Write a short product description for a Tokyo VPS.",
  "stream": false
}'

# OpenAI-compatible API (base URL http://127.0.0.1:11434/v1, any API key)
curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen2.5:7b","messages":[{"role":"user","content":"Hello"}]}'

Workflow tools such as Dify and n8n can use Ollama as a model provider; see Dify, n8n and AI agent server requirements.

Step 7: Tune Performance

• Keep the model loaded: by default a model is unloaded after 5 minutes idle. Set Environment="OLLAMA_KEEP_ALIVE=1h" in the override file to avoid slow first answers.
• Parallel requests: OLLAMA_NUM_PARALLEL=2 lets two requests share a loaded model; each extra slot needs more RAM.
• Context length: long documents need a larger context window, which also uses more memory. Increase it only when needed.
• Leave RAM free: if the server starts swapping, answers become extremely slow. Use ollama ps and free -h to check.

FAQ

Is a CPU-only server really usable?

Yes for internal tools, document Q&A, drafting, translation and batch jobs with 3B–8B models. For many concurrent users or large models, you need more cores and RAM, or GPU hardware.

Which model is best for Chinese?

The Qwen family is trained heavily on Chinese and English and is a good default. Try qwen2.5:7b first, then compare with other models on your own questions.

Does my data leave the server?

No. Ollama and Open WebUI process prompts and documents locally. Only downloading models and updates contacts the internet.

Still stuck after following these steps? Open a support ticket and the IMIDC 24/7 technical team will help. Include the server IP, OS version, CPU and RAM, the model you use and the full error output so we can pinpoint the issue faster.