Klippy📎

Paperclips Organized Your Documents Since 1867.
Now They Understand Them.

A text-centric multimodal RAG system with knowledge graph reasoning for sub-4B LLMs

---

🎯 What is Klippy?

Klippy is a local-first AI assistant that remembers everything. It combines:

• 🔍 Multimodal RAG ; Search across text, images, and audio with one query
• 🧠 Knowledge Graph Reasoning ; Neo4j-powered ontological reasoning for complex queries
• ⚡ Prompt Chaining ; Google ADK agents orchestrate multi-step reasoning pipelines
• 💾 Personal Memory ; Your data stays local, your assistant gets smarter

Architecture Philosophy: "Graph-as-Brain, LLM-as-Mouth" ; All reasoning is pre-computed via deterministic graph logic. The LLM only narrates the answer. This allows sub-4B models like Qwen2.5:3b to perform like much larger models.

---

✨ Features

🎨 Multimodal Ingestion 📝 Text ; TXT, MD, PDF, code files 🖼️ Images ; BLIP captioning + Tesseract OCR 🎵 Audio ; Whisper transcription 🔄 Differential Updates ; Skip unchanged files via content hashing	🔍 Unified Search Single vector space for ALL content (384-dim) One query searches text, images, and audio Filter by content type, source, or entity HNSW indexing for sub-millisecond search
🧠 Knowledge Graph Reasoning Entity extraction with spaCy NER Relationship mining (Subject → Predicate → Object) Multi-hop graph traversal Causal chain analysis Pre-computed reasoning chains for small LLMs	🤖 Agent Architecture (Google ADK) Orchestrator Agent ; Routes and coordinates Qdrant Agent ; Semantic search specialist Neo4j Agent ; Knowledge graph queries Prompt Chain Agent ; Multi-step reasoning pipeline
🖥️ Desktop App (Electron) Native Windows/macOS/Linux app Real-time chat interface Source file previews with images Confidence scores and entity badges	⚙️ Context-Aware Responses Ontological enrichment before RAG Smart context truncation for small LLMs Confidence scoring per reasoning step Source attribution with file paths

---

🏗️ Architecture

---

🛠️ Tech Stack

Category	Technologies	Purpose
Backend		REST API, validation, async I/O
Frontend		Desktop app, chat UI
Vector DB		Semantic search, HNSW indexing
Graph DB		Knowledge graph, Cypher queries
LLM		Local inference
Agents		Agent orchestration
Embeddings		Text embeddings (384-dim)
Vision		Image captioning, OCR
Speech		Speech-to-text
NLP		NER, relationship extraction
DevOps		Containers, fast packaging

---

🚀 Quick Start

Prerequisites

• Python 3.12+
• Docker Desktop (for Qdrant & Neo4j)
• Ollama (for local LLM)
• Node.js 18+ (for frontend)
• uv package manager (recommended)

1️⃣ Clone & Install

git clone https://github.com/Rayen-Hamza/Klippy.git
cd Klippy

# Install Python dependencies
uv sync

# Download spaCy model
uv run python -m spacy download en_core_web_sm

2️⃣ Start Databases

docker compose up -d

This starts:

• Qdrant on localhost:6333 (Vector DB)
• Neo4j on localhost:7474 (Graph DB, password: changeme)

3️⃣ Start Ollama

# In a separate terminal (or it may already be running)
ollama serve

# Pull the model (first time only)
ollama pull qwen2.5:3b

4️⃣ Start the API

uv run uvicorn app.main:app --host 0.0.0.0 --port 8000

📚 API docs: http://localhost:8000/docs

5️⃣ Start the Desktop App (Optional)

cd frontend
npm install
npm start

---

📡 API Reference

Ingestion

Endpoint	Method	Description
/ingest/text	POST	Upload text/PDF/markdown
/ingest/image	POST	Upload image → caption + OCR
/ingest/audio	POST	Upload audio → transcribe
/ingest/directory	POST	Batch ingest from path

Search

Endpoint	Method	Description
/search	POST	Unified search ; all modalities
/search/by-type/{type}	POST	Filter by text/image/audio
/search/filters/by-entity	GET	Find content by entity

Agents (Google ADK)

Endpoint	Method	Description
/agent/chat	POST	Chat with orchestrator agent
/agent/sessions	GET	List active sessions
/agent/agents	GET	List available agents

Reasoning

Endpoint	Method	Description
/reasoning/query	POST	Graph reasoning → LLM prompt
/reasoning/ingest	POST	Ingest document to graph

---

🧪 Testing

# Run all tests
uv run pytest

# Run with coverage
uv run pytest --cov=app

# Run specific test
uv run pytest tests/test_embeddings.py -v

---

📁 Project Structure

Klippy/
├── app/                          # FastAPI backend
│   ├── agents/                   # Google ADK agents
│   │   ├── orchestrator.py       # Root agent (router)
│   │   ├── qdrant_agent.py       # Vector search specialist
│   │   ├── neo4j_agent.py        # Knowledge graph specialist
│   │   └── prompt_chain.py       # Prompt chaining pipeline
│   ├── models/                   # Pydantic models
│   ├── routes/                   # API endpoints
│   ├── services/                 # Business logic
│   │   ├── embeddings/           # Embedding strategies
│   │   ├── processing/           # Text/Image/Audio processors
│   │   └── storage/              # Qdrant manager
│   └── config.py                 # Settings
├── frontend/                     # Electron desktop app
│   └── src/
│       ├── main/                 # Electron main process
│       └── renderer/             # React UI
├── tests/                        # pytest tests
├── docker-compose.yml            # Qdrant + Neo4j
└── pyproject.toml                # Python dependencies

---

🔧 Configuration

Create a .env file:

# Qdrant
QDRANT_HOST=localhost
QDRANT_PORT=6333

# Neo4j
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=changeme

# Ollama (Local LLM)
LLM_PROVIDER=ollama
LLM_MODEL=qwen2.5:3b
LLM_BASE_URL=http://localhost:11434/v1

# Processing
TEXT_CHUNK_SIZE=512
TEXT_CHUNK_OVERLAP=50
LOG_LEVEL=INFO

---

🏆 Hackathon Team

CrèmeTartinéDangereuse Built with 💔 and ☕

---

📜 License

MIT License ; see LICENSE for details.

---

_{⭐ Star this repo if you find it useful!}