-
Notifications
You must be signed in to change notification settings - Fork 3
Open
Labels
enhancementNew feature or requestNew feature or requestperformanceCan possibly improve performanceCan possibly improve performance
Description
Right now lookup using the Find(key) method for a single key isn't necessarily a significant performance bottleneck. However, when checking all keys, when len(keys) is large, it can be quite slow.
We should add a parallel FindMany method; still need to think more about the signature of the method to make it flexible for several uses, but here is a first sketch:
func (bb BBHash) FindMany(keys []uint64) []uint64 {
var wg sync.WaitGroup
keyChunks := slices.Chunk(keys, 1000)
a := make([][]uint64, 0, len(keys)/1000)
for chunk := range keyChunks {
wg.Add(1)
idx := make([]uint64, len(chunk))
a = append(a, idx)
go func() {
defer wg.Done()
for i, key := range chunk {
idx[i] = bb.Find(key)
}
}()
}
wg.Wait()
return slices.Flatten(a) // Flatten does not exist
}Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestperformanceCan possibly improve performanceCan possibly improve performance