This library aims to bring the fantastic LanceDB embedded multimodal vector search database to the BEAM. As LanceDB is written in Rust, this integration uses Rustler to map NIFs to the Lance Rust client.
The current focus of this project is to implement an Ecto Adapter so that it can plug into the broader Ecto ecosystem.
🚧 This project is very new, in active development, and is not published on Hex. 🚧
🚧 Feel free to fork, borrow from, or use at your own risk, but do expect the API to change. 🚧
- Return a DB Connection from LanceDB to Elixir
- List DB tables
- Create DB table from schema
- Drop DB tables
- Create DB table from data
- Read data back from table
- Hold peristent connection to table
- List items from table
- Query with SQL filter and limit
- Add items to existing table
- Update items in table
- Delete items from table
- Vector search
- Full text search
- Hybrid Vector + Full Text search
- Optimize table
- Add, drop, and alter table columns
- Implement an Ecto Adapter
- Implement an Ash DataLayer
- Implement table Cleanup/Compaction strategies
- Implement Lance embedding registry for OpenAI and Sentence Transformers (no need for users to generate own embeddings)
- Complete various partial implementations - indices, new column configs, data types, table optimization, etc
- More tests, benchmarks, and documentation
- Distribute with
rustler_precompiled
Thanks to:
- The good folks on the LanceDB discord for guidance and encouraging emojis
- The good folks on ElixirForum
- Rustler!
- Elixir Explorer
- arrow-elixir
- snowflake_arrow
- Polars
- This convo: jorgecarleitao/arrow2#1033