Quafka 🦆

Kafka/distributed commit log service in Go.

Quafka = Kafka + Go (sounds like a duck!) 🦆

Goals

• Implement Kafka in Go
• Protocol compatible with Kafka so Kafka clients and services work with Quafka ✅
• Make operating simpler
• Distribute a single binary
• Use Serf for discovery, Raft for consensus (no ZooKeeper dependency)
• Smarter configuration settings
  • Able to use percentages of disk space for retention policies rather than only bytes and time kept
  • Handling size configs when you change the number of partitions or add topics

Note: Kafka client compatibility has been verified with Sarama. See _examples/sarama/ for a working example.

What's Changed (from Jocko)

This is a fully modernized fork of Jocko with significant updates:

• ✨ Consumer Groups - Full consumer group protocol support (JoinGroup, SyncGroup, OffsetCommit, etc.)
• 🚀 Go 1.23 (was 1.12)
• 📦 All dependencies updated (sarama, raft, serf, uuid, etc.)
• 🔧 Bug fixes - Protocol decoder, leader election, panic handling
• 🧹 Code quality - Removed panics, gotos, debug prints
• ⚡ CI/CD - GitHub Actions with linting
• 📝 Package rename - Internal package renamed from jocko to quafka

👉 See CHANGELOG.md for the complete list of changes.

If you're migrating from Jocko, the CHANGELOG has a migration guide.

Status

• Producing
• Fetching
• Partition consensus and distribution
• Discovery
• Consumer Groups ✨
• Protocol APIs
  • Produce / Fetch
  • Metadata
  • Create / Delete Topics
  • Consumer Group (JoinGroup, SyncGroup, Heartbeat, LeaveGroup)
  • Offset Commit / Fetch
  • FindCoordinator, ListGroups, DescribeGroups
  • Transactions
  • ACLs
• API versioning (more versions to implement)
• Replication (first draft done)
• Security (SSL/TLS, SASL)

See FEATURES.md for a detailed feature checklist.

Project Layout

├── cmd/quafka     command to run a Quafka broker and manage topics
├── commitlog      low-level commit log implementation
├── quafka         broker, server, and core subsystems
│   ├── config     configuration
│   ├── fsm        finite state machine for Raft
│   ├── metadata   broker metadata
│   └── structs    data structures
├── log            logging utilities
├── mock           mocks for testing
├── protocol       Kafka protocol implementation
└── testutil       test utilities

Building

Prerequisites

• Go 1.23 or later

Local Build

git clone https://github.com/bodaay/quafka.git
cd quafka
make build

Docker

docker build -t quafka:latest .

Running

# Start a single broker
./cmd/quafka/quafka broker

# Start with custom configuration
./cmd/quafka/quafka broker --data-dir /tmp/quafka --broker-addr 127.0.0.1:9092

Contributing

See CONTRIBUTING for details on submitting patches and the contribution workflow.

Acknowledgments

This project is a modernized fork of Jocko, originally created by Travis Jeffery. Travis did the heavy lifting of implementing the Kafka protocol in Go and designing the distributed architecture using Raft and Serf. His work on Jocko and the accompanying blog posts laid the foundation for this project.

The articles he wrote are excellent resources for understanding distributed systems:

• Building a Kafka that doesn't depend on ZooKeeper
• How Kafka's storage internals work

License

Quafka is under the MIT license, see the LICENSE file for details.