Skip to content

Perf: Add a parallel FindMany method to speed up queries for a key #21

@meling

Description

@meling

Right now lookup using the Find(key) method for a single key isn't necessarily a significant performance bottleneck. However, when checking all keys, when len(keys) is large, it can be quite slow.

We should add a parallel FindMany method; still need to think more about the signature of the method to make it flexible for several uses, but here is a first sketch:

func (bb BBHash) FindMany(keys []uint64) []uint64 {
	var wg sync.WaitGroup
	keyChunks := slices.Chunk(keys, 1000)
	a := make([][]uint64, 0, len(keys)/1000)
	for chunk := range keyChunks {
		wg.Add(1)
		idx := make([]uint64, len(chunk))
		a = append(a, idx)
		go func() {
			defer wg.Done()
			for i, key := range chunk {
				idx[i] = bb.Find(key)
			}
		}()
	}
	wg.Wait()
	return slices.Flatten(a) // Flatten does not exist
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceCan possibly improve performance

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions