chatbot-confidential

Fully-contained private knowledgebase chatbot. Ask questions about your own documents without data privacy issues.

Installation and Usage

1. Download Docker Desktop and install it on your local machine.
2. Download ollama.ai and install it on your local machine
3. Download the chatbot-confidential compose.yaml (right click and Save As...)
4. Open your local command line terminal - Mac Instructions | Windows Instructions
5. Navigate (cd foldername) to the folder where you downloaded compose.yaml. Then type docker compose up in the command line terminal, and hit enter. (To stop, close the terminal window, or use Ctl-C)
6. Open http://localhost:8000/ in a browser on your machine

Features:

• runs entirely on LOCAL desktop machine: NO data in the cloud, NO API calls to the cloud
• includes data upload and processing of documents - supported file types: .pdf .docx .txt
• supports Apple Silicon Neural Engine (M1, M2, etc) acceleration and does NOT require an NVIDIA GPU
• uses Retrieval Augmentation (RAG) to find documents relevant to your chat question, and use that in answering
• uses Hypothetical Document Embeddings (HyDE) for improved document relevance
• also uses NEW question-generation approach for further improved document relevance (see my blog post)

Implementation:

• local LLM model: OpenOrca - Mistral - 7B run using ollama.ai
• local embedding model: gte-large run using ONNX Runtime
• uses pgvector for local vector database - using cosine similarity and HNSW
• uses Docker Compose
• Data is stored locally using Docker Desktop Volumes

Developer Setup

Instructions on developer setup for macOS

Pre-requisites:

• pyenv
• Python Poetry
• nvm

Python Developer Setup

pyenv install 3.11
pyenv local 3.11

poetry install

Install and initialize mkcert This enables local https via sanic

brew install mkcert
brew install nss

mkdir ~/dev-tls
cd ~/dev-tls
mkcert -install
mkcert example.com "*.example.com" example.test localhost 127.0.0.1 ::1

cp example.com+5-key.pem privkey.pem
cp example.com+5.pem fullchain.pem
chmod 0600 *.pem

Web Developer Setup

nvm install v18
nvm use v18

cd frontend_quasar_vue
npm install
npm run build

Develop

create a local .env file in the chatbot-confidential folder:

APP_DATA_PATH=/tmp/chatbot-data
APP_OLLAMA_HOST=0.0.0.0:11434
APP_POSTGRES_HOST=0.0.0.0:5432
APP_POSTGRES_USER=postgres
APP_POSTGRES_PASSWORD=notsoseekret
APP_POSTGRES_DB=postgres

and create the data path locally on your own machine

in the chatbot-confidential folder:

poetry run sanic server:app --dev --tls=`echo ~/dev-tls/`

in a new terminal window:

docker run -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=notsoseekret \
  -p 5432:5432 \
  -v postgres:/var/lib/postgresql/data \
  ankane/pgvector:v0.5.1

in a new terminal window, in the chatbot-confidential folder:

cd frontend_quasar_vue
npm run dev