How to build a RAG from scratch: practical guide for Ops

RAG (Retrieval-Augmented Generation) is an architectural pattern that combines two things: Retrieval: Find relevant information in your documents using semantic search ===

What is a RAG

RAG (Retrieval-Augmented Generation) is an architectural pattern that combines two things:

Retrieval (Retrieval): Search for relevant information in your documents using semantic search (vectors).
generation (Generation): Send that information as context to an LLM so that it generates a response based on real data.

Why does it matter? Because an LLM by itself "hallucinates" (invents data). With RAG, the model only responds with information that actually exists in your documents. It's the difference between asking someone who knows nothing about your infra vs. give it your runbooks and respond based on them.

Use case: Ops teams

Imagine being able to load all your internal documentation (runbooks, configurations, playbooks, postmortems) and ask in natural language:

"How do I restart the Kafka cluster in production?"
"What firewall rules do we have for the DMZ segment?"
"What was the workaround for incident INC-4523?"

That's exactly what we're going to build.

What are we going to ride?

ChromaDB as vector database (lightweight, persistent, in Docker)
Any LLM compatible with OpenAI format as a generation engine
File ingestion .md, .yaml, .conf, .sh of your servers
Simple CLI: python rag.py -q "tu pregunta"

For this proof of concept we will use the free APIs of Grok (xAI) and NVIDIA NIM, which do not require payment or deploy your own infrastructure. The code is prepared for you to connect to any LLM (Azure OpenAI, AWS Bedrock, a self-hosted model, etc.) by simply changing an environment variable.

For enterprise environments: At the end of the post I include recommendations on what options exist to deploy a private LLM with the security and compliance guarantees you need.

Prerequisites

Component	Minimum	Recommended
RAM	4GB	8GB+
Disk	5 GB free	10GB+
Docker	24.x+	27.x+
Docker Compose	v2.x	v2.x
Python	3.11+	3.13+

You will also need an API key from at least one of these providers:

Grok (xAI): Get in console.x.ai
NVIDIA NIM: Get in build.nvidia.com

Both offer enough free tier to follow this tutorial.

Architecture

TERRAFORM

┌─────────────────────────────────────────────────────┐
│                    Usuario                            │
│         python rag.py --provider nvidia              │
└──────────────────────┬──────────────────────────────┘
                       │
                       ▼
┌──────────────────────────────────────────────────────┐
│                  RAG Engine (Python)                   │
│                                                       │
│  1. Recibe pregunta                                   │
│  2. Genera embedding → consulta ChromaDB              │
│  3. Recupera documentos relevantes (top 5)            │
│  4. Construye prompt con contexto                     │
│  5. Envia al LLM via API                             │
│  6. Devuelve respuesta                                │
└───────┬──────────────────┬───────────────────────────┘
        │                  │
        ▼                  ▼
┌──────────────┐   ┌──────────────────────────┐
│   ChromaDB   │   │      LLM Provider        │
│  (Vectores)  │   │                          │
│  Docker      │   │ Grok API (xAI)           │
│              │   │ NVIDIA NIM               │
│              │   │ Azure OpenAI / Bedrock   │
└──────────────┘   └──────────────────────────┘

The key to this architecture is the separation between recovery logic (vectors) and generation logic (LLM). Changing LLM providers is just changing an environment variable.

Step 1: Project Structure

BASH

mkdir -p rag-ops/{data,chroma_data}
cd rag-ops

Final structure:

CODE

rag-ops/
├── docker-compose.yaml
├── requirements.txt
├── rag.py
├── ingest.py
├── config.py
├── .env
├── .gitignore
└── data/
    └── (tus ficheros .md, .yaml, .conf, .sh)

Create the .gitignore from the beginning:

BASH

echo -e ".env\n.venv/\n__pycache__/\nchroma_data/" > .gitignore

Step 2: Docker Compose - ChromaDB

We only need ChromaDB in Docker. The LLM is consumed via external API.

Create docker-compose.yaml:

YAML

services:
  chromadb:
    image: chromadb/chroma:latest
    container_name: rag-chromadb
    ports:
      - "8000:8000"
    volumes:
      - ./chroma_data:/chroma/chroma
    environment:
      - ANONYMIZED_TELEMETRY=FALSE
    restart: unless-stopped

Note: If you have port 8000 busy (for example with another service), change the mapping to "8001:8000" and adjust CHROMA_PORT in it .env to 8001. The internal port of the container is always 8000.

Lift service:

BASH

docker compose up -d

Verify that it is running:

BASH

curl http://localhost:8000/api/v2/heartbeat
# Respuesta esperada: {"nanosecond heartbeat":...}

Step 3: Get API Keys (for our PoC)

For this proof of concept we will use two free providers. In a real environment, you would replace these APIs with that of your corporate LLM (see final section).

Grok (xAI)

Go to console.x.ai
Create an account or log in
In the dashboard, generate an API key (start with xai-)
The free tier includes enough tokens for testing

Grok API supports OpenAI format, so we will use the same SDK with different base_url.

NVIDIA NIM

Go to build.nvidia.com
Create an NVIDIA account
Select a model with "Free Endpoint" (recommended: z-ai/glm4.7 either mistralai/mistral-nemotron)
Click "Get API Key" (start with nvapi-)

Important: Some NVIDIA NIM models have geographic restrictions. If you see "This NIM is unavailable in your location", try another model that has "Free Endpoint".

NVIDIA NIM also uses OpenAI-compatible format.

Step 4: Python Dependencies

Create requirements.txt:

TEXT

langchain>=0.3.25
langchain-huggingface>=1.0.0
langchain-openai>=0.3.12
langchain-text-splitters>=0.3.0
chromadb>=1.0.0
sentence-transformers>=4.0.0
python-dotenv>=1.0.0
rich>=14.0.0

Install in a virtualenv:

BASH

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Important: Always activate the virtualenv (source .venv/bin/activate) before running the scripts. If you see ModuleNotFoundError, you don't have venv active.

Step 5: Configuration

Create the file .env:

BASH

# Proveedor LLM: "grok" o "nvidia"
LLM_PROVIDER=nvidia

# Grok (xAI) - Compatible con formato OpenAI
GROK_API_KEY=xai-tu-clave-aqui
GROK_BASE_URL=https://api.x.ai/v1
GROK_MODEL=grok-3-mini

# NVIDIA NIM - Compatible con formato OpenAI
NVIDIA_API_KEY=nvapi-tu-clave-aqui
NVIDIA_BASE_URL=https://integrate.api.nvidia.com/v1
NVIDIA_MODEL=z-ai/glm4.7

# ChromaDB
CHROMA_HOST=localhost
CHROMA_PORT=8000
CHROMA_COLLECTION=ops-docs

# Embeddings (se ejecutan localmente, no requieren API)
EMBEDDING_MODEL=all-MiniLM-L6-v2

Security: Never go up .env to git. We already have it in .gitignore.

Create config.py:

PYTHON

import os
from dotenv import load_dotenv

load_dotenv()


class Config:
    # LLM Provider
    LLM_PROVIDER = os.getenv("LLM_PROVIDER", "grok")

    # Grok (xAI)
    GROK_API_KEY = os.getenv("GROK_API_KEY", "")
    GROK_BASE_URL = os.getenv("GROK_BASE_URL", "https://api.x.ai/v1")
    GROK_MODEL = os.getenv("GROK_MODEL", "grok-3-mini")

    # NVIDIA NIM
    NVIDIA_API_KEY = os.getenv("NVIDIA_API_KEY", "")
    NVIDIA_BASE_URL = os.getenv("NVIDIA_BASE_URL", "https://integrate.api.nvidia.com/v1")
    NVIDIA_MODEL = os.getenv("NVIDIA_MODEL", "z-ai/glm4.7")

    # ChromaDB
    CHROMA_HOST = os.getenv("CHROMA_HOST", "localhost")
    CHROMA_PORT = int(os.getenv("CHROMA_PORT", "8000"))
    CHROMA_COLLECTION = os.getenv("CHROMA_COLLECTION", "ops-docs")

    # Embeddings
    EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "all-MiniLM-L6-v2")

Step 6: Ingest Script

This script reads your documents, divides them into chunks, generates embeddings and stores them in ChromaDB.

Create ingest.py:

PYTHON

#!/usr/bin/env python3
"""Ingesta de documentos en ChromaDB para el RAG."""

import sys
from pathlib import Path

import chromadb
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from rich.console import Console
from rich.progress import track

from config import Config

console = Console()


def load_documents(data_dir: str) -> list[dict]:
    """Carga todos los ficheros soportados del directorio."""
    extensions = {".md", ".yaml", ".yml", ".conf", ".sh", ".txt", ".json", ".toml"}
    documents = []

    data_path = Path(data_dir)
    if not data_path.exists():
        console.print(f"[red]Error: directorio '{data_dir}' no existe[/red]")
        sys.exit(1)

    for filepath in data_path.rglob("*"):
        if filepath.suffix.lower() in extensions and filepath.is_file():
            try:
                content = filepath.read_text(encoding="utf-8")
                documents.append({
                    "content": content,
                    "metadata": {
                        "source": str(filepath.relative_to(data_path)),
                        "extension": filepath.suffix,
                        "filename": filepath.name,
                    }
                })
            except Exception as e:
                console.print(f"[yellow]Aviso: no se pudo leer {filepath}: {e}[/yellow]")

    return documents


def chunk_documents(documents: list[dict]) -> list[dict]:
    """Divide documentos en chunks manejables."""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        separators=["\n## ", "\n### ", "\n\n", "\n", " "],
    )

    chunks = []
    for doc in documents:
        splits = splitter.split_text(doc["content"])
        for i, split in enumerate(splits):
            chunks.append({
                "content": split,
                "metadata": {
                    **doc["metadata"],
                    "chunk_index": i,
                }
            })

    return chunks


def ingest(data_dir: str = "data"):
    """Pipeline principal de ingesta."""
    console.print("[bold blue]RAG Ops - Ingesta de documentos[/bold blue]\n")

    # 1. Cargar documentos
    console.print(f"Cargando documentos de [green]{data_dir}/[/green]...")
    documents = load_documents(data_dir)
    console.print(f"  Encontrados: [bold]{len(documents)}[/bold] ficheros\n")

    if not documents:
        console.print("[red]No se encontraron documentos para ingestar.[/red]")
        sys.exit(1)

    # 2. Dividir en chunks
    console.print("Dividiendo en chunks...")
    chunks = chunk_documents(documents)
    console.print(f"  Total chunks: [bold]{len(chunks)}[/bold]\n")

    # 3. Generar embeddings
    console.print(f"Generando embeddings con [green]{Config.EMBEDDING_MODEL}[/green]...")
    console.print("  (primera vez descarga el modelo ~80MB, luego es instantaneo)\n")
    embeddings = HuggingFaceEmbeddings(model_name=Config.EMBEDDING_MODEL)

    # 4. Almacenar en ChromaDB
    console.print(f"Conectando a ChromaDB en {Config.CHROMA_HOST}:{Config.CHROMA_PORT}...")
    client = chromadb.HttpClient(host=Config.CHROMA_HOST, port=Config.CHROMA_PORT)

    # Borrar coleccion existente si hay (re-ingesta limpia)
    try:
        client.delete_collection(Config.CHROMA_COLLECTION)
        console.print("  Coleccion anterior eliminada.")
    except Exception:
        pass

    collection = client.create_collection(
        name=Config.CHROMA_COLLECTION,
        metadata={"hnsw:space": "cosine"},
    )

    # 5. Insertar chunks
    console.print("\nInsertando chunks en ChromaDB...")
    batch_size = 100
    for i in track(range(0, len(chunks), batch_size), description="Ingesta"):
        batch = chunks[i:i + batch_size]
        texts = [c["content"] for c in batch]
        metadatas = [c["metadata"] for c in batch]
        ids = [f"chunk_{i + j}" for j in range(len(batch))]

        embs = embeddings.embed_documents(texts)

        collection.add(
            documents=texts,
            embeddings=embs,
            metadatas=metadatas,
            ids=ids,
        )

    console.print(f"\n[bold green]Ingesta completada![/bold green]")
    console.print(f"  Documentos: {len(documents)}")
    console.print(f"  Chunks almacenados: {len(chunks)}")
    console.print(f"  Coleccion: {Config.CHROMA_COLLECTION}")


if __name__ == "__main__":
    data_dir = sys.argv[1] if len(sys.argv) > 1 else "data"
    ingest(data_dir)

Step 7: The RAG Engine

Create rag.py:

PYTHON

#!/usr/bin/env python3
"""RAG Ops - Asistente de infraestructura con soporte multi-proveedor LLM."""

import argparse
import sys

import chromadb
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from rich.console import Console
from rich.markdown import Markdown
from rich.panel import Panel

from config import Config

console = Console()

SYSTEM_PROMPT = """Eres un asistente tecnico de infraestructura y operaciones.
Responde usando la informacion del contexto proporcionado.
Si no encuentras la respuesta en el contexto, di "No tengo informacion sobre eso en los documentos cargados."
Da respuestas detalladas y completas. Incluye ejemplos de configuracion, comandos y nombres de ficheros cuando sea relevante.
Estructura la respuesta con secciones si es necesario.
Responde en el mismo idioma en que te preguntan.

CONTEXTO:
{context}
"""


def get_llm(provider: str):
    """Factory que devuelve el LLM segun el proveedor.

    Tanto Grok como NVIDIA NIM son compatibles con el formato OpenAI,
    por lo que usamos ChatOpenAI con diferente base_url y api_key.
    Para anadir cualquier otro proveedor OpenAI-compatible, solo necesitas
    una nueva entrada con su base_url, api_key y model.
    """
    if provider == "grok":
        if not Config.GROK_API_KEY:
            console.print("[red]Error: GROK_API_KEY no configurada en .env[/red]")
            sys.exit(1)
        console.print(f"  LLM: Grok ({Config.GROK_MODEL})")
        return ChatOpenAI(
            api_key=Config.GROK_API_KEY,
            base_url=Config.GROK_BASE_URL,
            model=Config.GROK_MODEL,
            temperature=0.1,
            max_tokens=4096,
        )
    elif provider == "nvidia":
        if not Config.NVIDIA_API_KEY:
            console.print("[red]Error: NVIDIA_API_KEY no configurada en .env[/red]")
            sys.exit(1)
        console.print(f"  LLM: NVIDIA NIM ({Config.NVIDIA_MODEL})")
        return ChatOpenAI(
            api_key=Config.NVIDIA_API_KEY,
            base_url=Config.NVIDIA_BASE_URL,
            model=Config.NVIDIA_MODEL,
            temperature=0.1,
            max_tokens=4096,
            extra_body={
                "chat_template_kwargs": {
                    "enable_thinking": False,
                }
            },
        )
    else:
        console.print(f"[red]Proveedor '{provider}' no soportado. Usa 'grok' o 'nvidia'.[/red]")
        sys.exit(1)


def search_context(query: str, n_results: int = 5) -> str:
    """Busca documentos relevantes en ChromaDB."""
    embeddings = HuggingFaceEmbeddings(model_name=Config.EMBEDDING_MODEL)
    client = chromadb.HttpClient(host=Config.CHROMA_HOST, port=Config.CHROMA_PORT)

    collection = client.get_collection(Config.CHROMA_COLLECTION)
    query_embedding = embeddings.embed_query(query)

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
    )

    # Construir contexto con fuentes
    context_parts = []
    for doc, metadata in zip(results["documents"][0], results["metadatas"][0]):
        source = metadata.get("source", "desconocido")
        context_parts.append(f"[Fuente: {source}]\n{doc}")

    return "\n\n---\n\n".join(context_parts)


def ask(question: str, provider: str):
    """Pipeline principal: buscar contexto -> generar respuesta."""
    # 1. Buscar contexto relevante
    console.print("Buscando en documentos...")
    context = search_context(question)

    if not context:
        console.print("[yellow]No se encontraron documentos relevantes.[/yellow]")
        return

    # 2. Construir prompt
    prompt = ChatPromptTemplate.from_messages([
        ("system", SYSTEM_PROMPT),
        ("human", "{question}"),
    ])

    # 3. Obtener LLM
    llm = get_llm(provider)

    # 4. Ejecutar chain
    chain = prompt | llm | StrOutputParser()

    console.print("Generando respuesta...\n")
    response = chain.invoke({
        "context": context,
        "question": question,
    })

    # 5. Mostrar resultado
    console.print(Panel(Markdown(response), title="Respuesta", border_style="green"))


def interactive_mode(provider: str):
    """Modo interactivo: pregunta tras pregunta."""
    console.print(Panel(
        f"[bold]RAG Ops - Modo interactivo[/bold]\n"
        f"Proveedor: [green]{provider}[/green]\n"
        f"Escribe 'salir' o 'exit' para terminar.",
        border_style="blue",
    ))

    while True:
        try:
            question = console.input("\n[bold blue]Pregunta>[/bold blue] ").strip()
            if question.lower() in ("salir", "exit", "quit", "q"):
                console.print("[dim]Hasta luego.[/dim]")
                break
            if not question:
                continue
            ask(question, provider)
        except KeyboardInterrupt:
            console.print("\n[dim]Interrumpido.[/dim]")
            break


def main():
    parser = argparse.ArgumentParser(description="RAG Ops - Asistente de infraestructura")
    parser.add_argument(
        "--provider", "-p",
        choices=["grok", "nvidia"],
        default=Config.LLM_PROVIDER,
        help="Proveedor LLM (default: valor de .env)",
    )
    parser.add_argument(
        "--question", "-q",
        type=str,
        help="Pregunta directa (sin modo interactivo)",
    )
    args = parser.parse_args()

    if args.question:
        ask(args.question, args.provider)
    else:
        interactive_mode(args.provider)


if __name__ == "__main__":
    main()

Step 8: Test the system

8.1 Add documents

Copy configuration or documentation files to data/. The more documentation you get, the better the answers will be:

BASH

# Ejemplos: configs, scripts, manuales, runbooks
cp /etc/nginx/nginx.conf data/
cp /etc/nginx/sites-available/* data/
cp /path/to/runbooks/*.md data/
cp /path/to/ansible/playbooks/*.yaml data/

Important: The RAG is only as good as the documentation you put into it. With a single default configuration file, the responses will be poor. Put real and detailed documentation to get useful answers.

8.2 Run ingestion

BASH

source .venv/bin/activate
python ingest.py data/

Expected output:

CODE

RAG Ops - Ingesta de documentos

Cargando documentos de data/...
  Encontrados: 45 ficheros

Dividiendo en chunks...
  Total chunks: 892

Generando embeddings con all-MiniLM-L6-v2...
  (primera vez descarga el modelo ~80MB, luego es instantaneo)

Conectando a ChromaDB en localhost:8000...

Insertando chunks en ChromaDB...
Ingesta ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%

Ingesta completada!
  Documentos: 45
  Chunks almacenados: 892
  Coleccion: ops-docs

The first run downloads the embeddings model (~80MB). The following are instantaneous because they are cached in ~/.cache/huggingface/.

8.3 Consult

BASH

# Pregunta directa
python rag.py --provider nvidia -q "Que reglas protegen contra SQL injection"

# Modo interactivo
python rag.py --provider nvidia

8.4 Change provider

BASH

# Con Grok
python rag.py --provider grok -q "Como configuro SSL en nginx"

# Con NVIDIA NIM
python rag.py --provider nvidia -q "Como configuro SSL en nginx"

Step 9: Add a new LLM provider

The architecture is prepared to add any provider compatible with the OpenAI format. The pattern is always the same: ChatOpenAI + base_url + api_key + model.

Example: Azure OpenAI

Add the variables to .env:

BASH

AZURE_OPENAI_API_KEY=tu-clave
AZURE_OPENAI_ENDPOINT=https://tu-recurso.openai.azure.com
AZURE_OPENAI_MODEL=gpt-4o
AZURE_API_VERSION=2024-10-21

Add the configuration in config.py:

PYTHON

    # Azure OpenAI
    AZURE_OPENAI_API_KEY = os.getenv("AZURE_OPENAI_API_KEY", "")
    AZURE_OPENAI_ENDPOINT = os.getenv("AZURE_OPENAI_ENDPOINT", "")
    AZURE_OPENAI_MODEL = os.getenv("AZURE_OPENAI_MODEL", "gpt-4o")
    AZURE_API_VERSION = os.getenv("AZURE_API_VERSION", "2024-10-21")

Add the case in get_llm() of rag.py:

PYTHON

elif provider == "azure":
    from langchain_openai import AzureChatOpenAI
    return AzureChatOpenAI(
        api_key=Config.AZURE_OPENAI_API_KEY,
        azure_endpoint=Config.AZURE_OPENAI_ENDPOINT,
        azure_deployment=Config.AZURE_OPENAI_MODEL,
        api_version=Config.AZURE_API_VERSION,
        temperature=0.1,
        max_tokens=4096,
    )

Duck "azure" to choices in the argparser.

The same pattern applies to AWS Bedrock, GCP Vertex AI, or any self-hosted LLM that exposes an OpenAI-compatible API (vLLM, Ollama, LiteLLM, etc.).

Step 10: Recommendations for enterprise environments

In a business environment, you should not use public APIs with sensitive internal data. Recommended options:

Option A: LLM managed in private cloud

Supplier	Service	Advantage
Azure	Azure OpenAI Service	Data does not leave your tenant, SLA 99.9%, SOC2/ISO compliance
Azure	Azure AI Foundry	Multi-model catalog (OpenAI, Llama, Mistral), integrated MLOps
AWS	Amazon Bedrock	Multi-model (Claude, Llama, Mistral), VPC endpoints
GCP	Vertex AI	Gemini, open-source models, integration with GKE

These services ensure that your data They are not used to train models and offer network isolation (Private Endpoints / VPC).

Option B: LLM self-hosted on your infrastructure

For maximum privacy, deploy the model in your own cluster:

vLLM on Kubernetes with GPUs (A100/H100) — exposes OpenAI-compatible API
NVIDIA NIM containers in your on-premise infrastructure
Ollama on a dedicated server with GPU for small teams

Option C: Hybrid

Embeddings: Always local (does not send data anywhere)
LLM for sensitive data: Azure OpenAI / Bedrock with Private Endpoint
LLM for public data: Direct API (Grok, NVIDIA NIM) for cost/speed

Minimum recommended security

Never send credentials, tokens or secrets to the LLM — filter before eating
Use environment variables or a secret manager (Vault, AWS Secrets Manager)
Audit the logs that is sent to the API
Implement RBAC if multiple teams use the RAG
Encrypt data at rest in ChromaDB (volume encryption)

Troubleshooting

ChromaDB not connecting

BASH

# Verificar que esta corriendo
docker logs rag-chromadb

# Probar conexion (ChromaDB v2+)
curl http://localhost:8000/api/v2/heartbeat

If you see Connection reset by peer, verify that the port mapping is correct on docker-compose.yaml. The container always listens on 8000 internally. If you map to another external port (e.g. 8001:8000), adjust CHROMA_PORT in .env.

Authentication error in Grok

CODE

openai.AuthenticationError: Incorrect API key provided

Verify that the key in .env start with xai- and is active in console.x.ai.

Authentication or geolocation error in NVIDIA NIM

CODE

openai.AuthenticationError: ...
# o
This NIM is unavailable in your location

Verify that the key begins with nvapi-
If there is geoblocking, change to another model with "Free Endpoint" in build.nvidia.com
Recommended models without restriction: z-ai/glm4.7, mistralai/mistral-nemotron

Empty or very short answers

If the LLM returns empty responses, it may be that the model uses "thinking mode" which puts the content in a separate field. The solution is to deactivate thinking in extra_body:

PYTHON

extra_body={
    "chat_template_kwargs": {
        "enable_thinking": False,
    }
}

If the answers are too short, increase max_tokens in the LLM configuration (recommended: 4096).

Poor or generic answers

The RAG is only as good as the documentation you put into it. If the answers are poor:

Add more relevant documentation in data/ — not only configs, but also manuals, runbooks, postmortems
Re-run the ingestion (python ingest.py data) after adding documents
Check how many chunks you have: few chunks = little context = poor answers

ModuleNotFoundError

TERRAFORM

ModuleNotFoundError: No module named 'langchain_huggingface'

Make sure you have virtualenv active:

BASH

source .venv/bin/activate
python rag.py ...

Next steps

Web interface: Add Streamlit or Gradio as a visual frontend
automatic intake: Watcher that re-ingests when files change
Multi-collection: Separate documents by project or environment
Assessment: Measure quality of responses with RAGAS
Record: Implement conversational memory for chained questions

Resources

:wq!

What is a RAG

Use case: Ops teams

What are we going to ride?

Prerequisites

Architecture

Step 1: Project Structure

Step 2: Docker Compose - ChromaDB

Step 3: Get API Keys (for our PoC)

Grok (xAI)

NVIDIA NIM

Step 4: Python Dependencies

Step 5: Configuration

Step 6: Ingest Script

Step 7: The RAG Engine

Step 8: Test the system

8.1 Add documents

8.2 Run ingestion

8.3 Consult

8.4 Change provider

Step 9: Add a new LLM provider

Example: Azure OpenAI

Step 10: Recommendations for enterprise environments

Option A: LLM managed in private cloud

Option B: LLM self-hosted on your infrastructure

Option C: Hybrid

Minimum recommended security

Troubleshooting

ChromaDB not connecting

Authentication error in Grok

Authentication or geolocation error in NVIDIA NIM

Empty or very short answers

Poor or generic answers

ModuleNotFoundError

Next steps

Resources

Comments