Build a Private AI Knowledge Base Chatbot with AnythingLLM and Ollama (RAG) on Your Own Server

A general chatbot does not know your products, prices, procedures or internal documents, and pasting them into a public AI service is often not acceptable. With retrieval-augmented generation (RAG), the assistant first searches your own documents and answers from what it finds, with citations. This guide sets up AnythingLLM with a local model served by Ollama on your own server: install, connect the language and embedding models, upload documents into workspaces, configure answers that stay within your documents, add users and publish it over HTTPS. For planning a larger deployment (region, vector database, data protection), see building a private AI support knowledge base on a dedicated server.

Step 1: Prepare the Models

RAG uses two models: an embedding model that turns document passages and questions into vectors for searching, and a chat model that writes the answer. Both run in Ollama.

ollama pull nomic-embed-text      # embedding model (small and fast on CPU)
ollama pull qwen2.5:7b            # chat model; good with Chinese and English
ollama list

# AnythingLLM runs in a container, so Ollama must accept connections from Docker
# (OLLAMA_HOST=0.0.0.0 plus the firewall rules from the Ollama guide). Check:
curl -s http://172.17.0.1:11434/api/tags | head -c 300; echo

A server with 8 or more cores and 16 GB of RAM runs this combination comfortably for a small team. Embedding documents uses CPU for a while when you upload many files; answering uses the chat model.

Step 2: Start AnythingLLM

AnythingLLM runs in Docker (see our Docker installation guide).

export STORAGE_LOCATION=/opt/anythingllm
mkdir -p $STORAGE_LOCATION && touch "$STORAGE_LOCATION/.env"
chown -R 1000:1000 $STORAGE_LOCATION      # the container runs as user ID 1000

docker run -d --name anythingllm --restart always \
  -p 127.0.0.1:3002:3001 \
  --cap-add SYS_ADMIN \
  --add-host=host.docker.internal:host-gateway \
  -v ${STORAGE_LOCATION}:/app/server/storage \
  -v ${STORAGE_LOCATION}/.env:/app/server/.env \
  -e STORAGE_DIR="/app/server/storage" \
  mintplexlabs/anythingllm

docker logs -f anythingllm      # wait until the server reports it is listening

The web interface listens on port 3001 inside the container; it is mapped to 127.0.0.1:3002 on the host so it is only reachable through the HTTPS proxy in Step 3.

Step 3: Publish It over HTTPS

Point a subdomain such as kb.example.com to the server, get a certificate (see free SSL with Certbot or acme.sh) and proxy to the container:

server {
    listen 443 ssl;
    http2 on;
    server_name kb.example.com;

    ssl_certificate     /etc/letsencrypt/live/kb.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/kb.example.com/privkey.pem;
    client_max_body_size 100m;          # document uploads

    location / {
        proxy_pass         http://127.0.0.1:3002;
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection "upgrade";
        proxy_set_header   Host $host;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_read_timeout 600s;
    }
}

Step 4: Connect the Models

Open https://kb.example.com and complete the setup (you can change everything later under Settings):

• LLM provider: Ollama, base URL http://host.docker.internal:11434, chat model qwen2.5:7b.
• Embedding provider: Ollama with nomic-embed-text (or AnythingLLM's built-in embedder).
• Vector database: the built-in LanceDB; no extra setup needed.

Step 5: Secure It with Users

Before you upload anything, enable multi-user mode in the security settings and set the administrator password. Then create an account for each colleague with the Default role, so they can chat with the workspaces you give them but cannot change settings or see other workspaces.

Step 6: Create Workspaces and Upload Documents

A workspace is a separate knowledge base with its own documents and settings, for example "Product catalogue", "Support procedures" or "HR policies".

1. Create a workspace and open its upload dialog.
2. Upload PDF, Word, text, Markdown or CSV files, or fetch a web page by URL.
3. Move the uploaded files into the workspace and choose Save and Embed. AnythingLLM splits them into passages and stores their vectors.

Clean, text-based documents give the best results. Scanned PDFs need OCR first; long documents with clear headings are easier to search than one huge file with everything.

Step 7: Keep Answers Within Your Documents

In the workspace settings:

• Set the chat mode to Query: the assistant answers only when it finds relevant passages and otherwise says it has no information, instead of guessing.
• Write a system prompt, for example: "You are the internal assistant of ACME. Answer only from the provided documents, cite the source, and reply in the language of the question."
• Adjust the number of passages used per answer and the similarity threshold if answers miss relevant facts or include unrelated ones.

Each answer shows citations; click them to check which document passage was used.

Step 8: Maintain the Knowledge Base

When a document changes, remove the old version from the workspace and upload the new one, so outdated prices or procedures are not quoted. Back up /opt/anythingllm regularly; it contains the documents, vectors, users and chat history. AnythingLLM also offers an API and an embeddable chat widget if you later want to use a workspace from your own website or applications.

FAQ

The assistant gives answers that are not in my documents

Switch the workspace to Query mode, tighten the system prompt, and check the citations. If the right passage is never found, split the document into smaller files with clear titles, or try a larger chat model.

Answers are slow

Most of the time is the chat model generating text on CPU. Use a smaller model for quick lookups, reduce the number of passages per answer, and keep the model loaded (see the performance tips in the Ollama guide).

Can I use a cloud AI API instead of Ollama?

Yes, AnythingLLM supports many providers. Keep in mind that your documents and questions are then sent to that provider, and use only providers and regions that their terms allow. For confidential material, the local setup in this guide keeps everything on your server.

Still stuck after following these steps? Open a support ticket and the IMIDC 24/7 technical team will help. Include the server IP, OS version, CPU and RAM, the output of docker logs anythingllm and the full error message so we can pinpoint the issue faster.