Intermittent errors creating a collection #187
Description
Hi, I am getting intermittent, inconsistent errors creating a collection. Any insight or assistance appreciated. Thank you in advance for your assistance!
Update: The issue seems to go away when I change the shard number from 4 to 2.
Issues
When I first run the below script, when the collection does not exist, I get Error: Error in the response: The operation was cancelled Timeout expired. Despite that, the qdrant_storage/MY_COLLECTION folder exists, and it shows up in the dashboard.
When I run it again, I get
collection exists, deleting...
Error: Error in the response: The operation was cancelled Timeout expired
I have also encountered issues where the client does not think the collection exists (so does not try deleting it), but when it tries to create it, I get errors like
Error in the response: Client specified an invalid argument Wrong input: Can't create collection with name MY_COLLECTION. Collection data already exists at ./storage/collections/MY_COLLECTION
Sometimes, it works fine, but it's unclear what conditions lead to success.
Code
main.rs
use anyhow::{Context, Result};
use qdrant_client::qdrant::{
CreateCollectionBuilder, Datatype, Distance, OptimizersConfigDiffBuilder, VectorParamsBuilder,
};
use qdrant_client::Qdrant;
const QDRANT_CONNECTION_URL: &str = "http://localhost:6334";
#[tokio::main]
async fn main() -> Result<()> {
let client = Qdrant::from_url(QDRANT_CONNECTION_URL)
.build()
.context("Failed to build Qdrant Client")?;
create_collection(&client, "MY_COLLECTION").await?;
Ok(())
}
pub async fn create_collection(client: &Qdrant, name: &str) -> Result<()> {
if client.collection_exists(name).await? {
println!("collection exists, deleting...");
client.delete_collection(name).await?;
}
client
.create_collection(
CreateCollectionBuilder::new(name)
.vectors_config(
VectorParamsBuilder::new(1280, Distance::Cosine)
.datatype(Datatype::Float16)
.on_disk(true),
)
.optimizers_config(OptimizersConfigDiffBuilder::default().indexing_threshold(0))
.shard_number(4),
)
.await?;
Ok(())
}Cargo.toml
[package]
name = "upload-issue"
version = "0.1.0"
edition = "2021"
[dependencies]
anyhow = "1.0.83"
qdrant-client = "1.11.2"
tokio = { version = "1.37.0", features = ["rt-multi-thread"] }