"recall_at_2": 0.162 for "sentence-transformers/all-MiniLM-L6-v2" and "LIMITSmallRetrieval"

[michaelwechner]

Hi

I have been running MTEB for "sentence-transformers/all-MiniLM-L6-v2" and "LIMITSmallRetrieval" and receive

"recall_at_2": 0.162

I have been running my own code using Apache Lucene Vector Search Implementation for "sentence-transformers/all-MiniLM-L6-v2" and "LIMITSmallRetrieval" and receive

"recall_at_2": 0.129

or when I run it a second time, then I receive

"recall_at_2": 0.134

Apache Lucene is using HNSW graph implementation https://lucene.apache.org/core/9_1_0/core/org/apache/lucene/util/hnsw/HnswGraph.html which quite probably has an effect on the search results, and therefore recall@2 is quite probably different than MTEB, but it would be good to compare this with the results of others.

I could not find anything on https://huggingface.co/spaces/mteb/leaderboard or otherwise.

Where could I find other results to compare my own implementation(s)?

[michaelwechner]

Btw, I have tested my own implementation also with Apache Lucene Keyword Search ("BM25") and get

"recall_at_2": 1

and also debugged it for a few individual queries to verify that the algorithm works really fine, which really seems to be the case.

[orionw]

Hi @michaelwechner and thanks for the interest!

That result makes sense to me. LIMIT is much easier for lexical models - my BM25 implementation with BM25s got high scores and a version of TF-IDF with the right vocab got 100% (footnote 9).

Glad to see you're able to find similar results!

[michaelwechner]

Hi @orionw thanks for your feedback!

Glad to hear, that I am not completely off :-)

Re footnote 9, I guess you are refering to your paper https://arxiv.org/pdf/2508.21038

"We also evaluate
three non-single vector models to show the distinction: BM25 [Robertson et al., 1995, Lù, 2024],
gte-ModernColBERT [Chaffin, 2025, Chaffin and Sourty, 2024], and a token-wise TF-IDF."

"This model turns each unique item into a token and then does TF-IDF. We build it to show that it gets 100% on all tasks
(as it reverse engineers our dataset construction) and thus we do not include it in future charts"