Replik

Your Interactive Voice + Visual AI Clone

An immersive dark-mode web application that creates an animated AI clone of you using:

• Voice cloning (Fish Audio API)
• Visual representation with audio-reactive waveforms
• Personality modeling (Claude API)
• Long-term memory (ChromaDB)
• Optional autonomous deployment (Fetch.ai Agentverse)

Try it live: replik.tech

---

Features

Voice Cloning

Record 20 seconds of audio to create a personalized voice model using Fish Audio's state-of-the-art TTS technology.

Visual Clone

Capture 5 selfies (front, left, right, up, down) to generate a ghostly outline of your face rendered with live audio waveforms.

Personality Model

Share stories, habits, and reactions to build a personality profile powered by Claude that makes your clone authentically you.

Vector Memory

All conversations and contexts are stored in ChromaDB, enabling your clone to recall past interactions and maintain context.

Autonomous Agent (Optional)

Deploy your clone to Fetch.ai's Agentverse where it can operate independently, interact with other agents, and persist beyond your session.

Minecraft Integration

Bring your digital twins into Minecraft with full voice support! Import twins from the web app, spawn them as NPCs, and chat with them in-game. Supports both command-based (MVP) and GUI-based (Advanced) modes.

See MINECRAFT_SETUP.md for full guide.

---

Documentation

For detailed setup guides, architecture documentation, and implementation notes, see the docs/ folder.

---

Tech Stack

Frontend

• Next.js 14 (App Router)
• React 18 + TypeScript
• Tailwind CSS (dark mode, custom neon palette)
• Framer Motion (animations)
• Web Audio API (waveform visualization)

Backend

• Next.js API Routes (serverless functions)
• Prisma (ORM)
• SQLite (development database)

AI & APIs

• Fish Audio - Voice cloning & TTS
• Anthropic Claude - Personality generation & conversation
• ChromaDB - Vector memory storage
• Fetch.ai - Autonomous agent deployment (optional)

---

Quick Start

Prerequisites

• Node.js 18+ and npm/yarn
• (Optional) ChromaDB server running locally or remotely

1. Clone & Install

git clone <your-repo-url>
cd replik
npm install

2. Environment Setup

Create a .env file in the root directory:

# Required APIs
FISH_AUDIO_API_KEY=your_fish_audio_api_key_here
ANTHROPIC_API_KEY=your_anthropic_api_key_here

# Database
DATABASE_URL="file:./dev.db"

# Optional: ChromaDB (defaults to localhost:8000)
CHROMA_HOST=localhost
CHROMA_PORT=8000

# Optional: Fetch.ai
FETCH_AI_API_KEY=your_fetch_ai_api_key_here

# Optional: Base URL for production
NEXT_PUBLIC_BASE_URL=http://localhost:3000

3. Database Setup

npx prisma generate
npx prisma db push

4. (Optional) Start ChromaDB

If you want persistent vector memory:

# Using Docker
docker run -p 8000:8000 chromadb/chroma

# Or install locally
pip install chromadb
chroma run --host localhost --port 8000

5. Run Development Server

npm run dev

Open http://localhost:3000 in your browser.

---

API Keys Setup

Fish Audio API

1. Sign up at fish.audio
2. Navigate to API settings
3. Generate API key
4. Add to .env as FISH_AUDIO_API_KEY

Documentation: https://fish.audio/docs

Anthropic Claude API

1. Sign up at console.anthropic.com
2. Generate API key
3. Add to .env as ANTHROPIC_API_KEY

Documentation: https://docs.anthropic.com/

ChromaDB (Optional but Recommended)

• Local: Run chroma run after installing via pip
• Cloud: Use Chroma Cloud or deploy your own instance

Documentation: https://docs.trychroma.com/

Fetch.ai Agentverse (Optional)

1. Sign up at fetch.ai
2. Create Agentverse account
3. Generate API credentials
4. Add to .env as FETCH_AI_API_KEY

Documentation: https://fetch.ai/docs

---

Project Structure

replik/
├── app/
│   ├── api/                    # API routes
│   │   ├── upload/            # Media upload handler
│   │   ├── voice-clone/       # Fish Audio integration
│   │   ├── personality/       # Claude personality generation
│   │   ├── memory/            # ChromaDB memory operations
│   │   ├── speak/             # Generate AI responses
│   │   ├── face-data/         # Face outline extraction
│   │   └── fetch-agent/       # Fetch.ai agent deployment
│   ├── globals.css            # Global styles
│   ├── layout.tsx             # Root layout
│   └── page.tsx               # Main page
├── components/
│   ├── Recorder.tsx           # Audio recording interface
│   ├── Uploader.tsx           # Photo & context upload
│   ├── WaveformCanvas.tsx     # Audio-reactive visualization
│   └── CloneChat.tsx          # Chat interface
├── lib/
│   ├── prisma.ts              # Prisma client
│   ├── api-clients.ts         # API client configurations
│   ├── audio-utils.ts         # Audio processing utilities
│   ├── face-utils.ts          # Face detection utilities
│   └── hooks/                 # Custom React hooks
│       ├── useAudioRecorder.ts
│       └── useCamera.ts
├── prisma/
│   └── schema.prisma          # Database schema
├── types/
│   └── index.ts               # TypeScript types
├── public/
│   └── uploads/               # User uploaded media
├── .env.example               # Environment template
├── package.json
├── tsconfig.json
├── tailwind.config.ts         # Tailwind configuration
└── next.config.js

---

Visual Design

Color Palette

• Background: #0a0a0f (dark-bg)
• Surface: #12121a (dark-surface)
• Neon Blue: #00d9ff
• Neon Cyan: #00fff5
• Neon Purple: #bf00ff
• Neon Pink: #ff00bf

Animation Style

• Ghost in the Shell / Daft Punk aesthetic
• Pulsing waveform outlines
• Smooth glowing transitions
• Minimalist dark UI with neon accents

---

User Flow

1. Record Voice (20 seconds)

   • Visualize audio levels in real-time
   • Auto-stop after duration

2. Capture Photos (5 angles)

   • Use webcam or upload files
   • Front, Left, Right, Up, Down views

3. Share Context

   • Stories, habits, reactions
   • Build personality model

4. Chat with Clone

   • Text conversation
   • AI-generated voice responses
   • Audio-reactive visual representation
   • Memory-enhanced interactions

---

Development Notes

Mock Modes

The app gracefully degrades if API keys are missing:

• Fish Audio: Uses mock voice model IDs, skips TTS
• Claude: Returns basic personality template
• ChromaDB: Falls back to database-only storage
• Fetch.ai: Disables agent deployment

Face Detection

Currently uses mock elliptical outlines. For production:

1. Install face-api.js or opencv4nodejs
2. Implement landmark detection in /app/api/face-data/route.ts
3. See comments in lib/face-utils.ts for integration guide

Database

• Development: SQLite (file-based)
• Production: Consider PostgreSQL with Prisma

---

Deployment

Vercel (Recommended)

npm run build
vercel deploy

Environment Variables: Add all .env variables in Vercel dashboard

Docker

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npx prisma generate
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]

ChromaDB Deployment

For production, deploy ChromaDB separately:

• Use Chroma Cloud
• Deploy on Railway/Render
• Self-host with Docker

---

Troubleshooting

Audio Recording Issues

• Firefox: May require HTTPS or localhost
• Safari: Check microphone permissions in System Preferences

Camera Access

• Ensure HTTPS (or localhost for development)
• Check browser permissions

ChromaDB Connection

• Verify server is running: curl http://localhost:8000/api/v1/heartbeat
• Check firewall settings

API Errors

• Verify API keys are correct
• Check rate limits
• Review console logs for details

---

Future Enhancements

• Multi-language voice cloning
• 3D face model rendering
• Real-time conversation mode
• Mobile app (React Native)
• Group clone interactions
• Voice modulation controls
• Advanced face detection with expressions
• Cloud-based persistent agents

---

License

MIT License

---

Contributing

Contributions are welcome!

1. Fork the repository
2. Create a feature branch
3. Submit a pull request

---

Acknowledgments

• Fish Audio - Voice cloning technology
• Anthropic - Claude API
• Chroma - Vector database
• Fetch.ai - Agent infrastructure

---

Support

For issues or questions:

• Open a GitHub issue
• Check API documentation
• Review troubleshooting section