Build a Private AI Assistant for Your Business Documents This Weekend

Your business already has a knowledge base — it's just scattered across PDFs, folders, and your own memory. Price lists, installation manuals, warranty documents, service SOPs, supplier catalogs. Every time a new employee asks a question your team has already answered a dozen times, that's a document that should be findable in seconds.

This weekend you can build a private AI assistant that reads all of those files and answers questions about them in plain English. It runs on your own computer, costs nothing, and keeps your data completely off the internet. Here's how.

What this actually does (no jargon)

The technique is called retrieval-augmented generation, or RAG. The plain-English version: you upload documents to a local index, and when you ask a question, the system finds the relevant passages and feeds them to an AI model, which writes you a clear answer from the source material. You're not asking an AI to guess from its training data — you're asking it to read your files and tell you what they say.

A few questions a well-loaded system can answer reliably:

"What's the warranty coverage on the Rheem water heaters we install?"
"What's our pricing for a 200A panel upgrade?"
"What are the winterization steps for Hunter irrigation systems?"
"What's our policy on after-hours call-out fees?"

For questions where the answer is already written down somewhere, this approach is accurate and fast. For open-ended judgment calls or real-time data, it's not the right tool.

The local route: AnythingLLM + Ollama (free, nothing leaves your machine)

You need two pieces of software, both free and open-source:

Ollama — runs AI models locally on your machine
AnythingLLM — the interface that connects to Ollama, ingests your documents, and gives you a chat window to query them

Minimum hardware: 8 GB of RAM and a modern Mac (M1 or newer) or Windows PC. 16 GB of RAM makes it noticeably faster. No GPU required, though a GPU speeds things up considerably.

Setup steps (2–3 hours including testing):

Install Ollama from ollama.com. It runs as a background service — no window appears, but it's ready.
Pull two models from your terminal. First, a chat model: ollama pull llama3.1 (about 4.7 GB — the 8B parameter version). Second, an embedding model that converts your documents into a searchable index: ollama pull nomic-embed-text (274 MB). The nomic-embed-text model is the standard choice for local RAG — it outperforms OpenAI's older embedding models on retrieval tasks and runs on any hardware, including CPU-only machines, according to Morph's 2026 Ollama model ranking. If you only have 8 GB of RAM, substitute ollama pull llama3.2:3b (2 GB) for the chat model — smaller but still capable for document Q&A.
Install AnythingLLM Desktop from anythingllm.com. On first launch, select Ollama as your LLM provider, enter http://localhost:11434 as the base URL, and pick llama3.1 (or whichever model you pulled). Then set the embedder to Ollama with nomic-embed-text.
Create a workspace — think of it as a folder for a topic area — and drag in your first documents. AnythingLLM supports PDF, DOCX, TXT, and more. Processing happens locally; nothing is sent over the internet. The full source code is on GitHub if you want to verify what the software does.
Test with real questions. Ask something you know the answer to. If the response is off, check that the document was fully processed (there's a status indicator per file in the sidebar).

AnythingLLM also supports multiple workspaces, so you can keep "Customer FAQ" separate from "Internal SOPs" — different document sets, same interface. Staff can use the same setup with role-based access once you move to the Docker version.

What documents to upload first

Start with the 5–10 documents your team reaches for most often. Don't try to index everything at once.

Your current price list or rate sheet — turns quote questions into instant lookups
Manufacturer warranty sheets for the products you install most frequently
1–3 installation or service manuals for your most common job types
Your service area and call-out policy — what you cover, what you don't, after-hours rates
Employee onboarding or SOPs — the documents new staff always ask about

The AI is only as good as what it can find. A well-structured PDF with clear section headings produces much better answers than a scan of a handwritten price list. If you notice the system fumbling a certain type of question, the fix is usually improving the source document — which is itself a useful audit of how clearly your business knowledge is written down.

The cloud alternative: Chatbase

If you'd rather not run local software, Chatbase is the no-install option. You upload your PDFs or paste in a website URL, and it builds a chatbot you can embed on your website or query directly in a dashboard. Paid plans start at $19/month for 2,000 messages. The free tier exists but caps at 50 messages/month and automatically deletes inactive bots after 14 days — useful for testing, not for production.

The tradeoff: Chatbase sends your documents to its servers and to the underlying model provider (OpenAI by default). For a public FAQ, that's fine. For internal pricing, contracts, or proprietary SOPs, the local AnythingLLM route keeps that data under your control.

What to do this weekend

Pick five documents your staff asks about most frequently.
Install Ollama and pull llama3.1 (or llama3.2:3b on 8 GB RAM) and nomic-embed-text.
Install AnythingLLM Desktop, configure it to point at Ollama, and upload your five docs.
Run 10 real test questions — things you already know the answers to — and note what the system gets right and what it misses.
For anything it fumbles, improve the source document and re-upload. Clearer inputs produce better answers.

By Sunday afternoon you'll have a working prototype. By the following week, your team will be using it to answer routine questions without interrupting you. The payoff is proportional to how well your business knowledge is documented — which is, itself, a useful thing to find out.