Filter callback interface for on-the-fly KNN filtering #128
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces a filter callback mechanism that allows external code to pass filter callbacks to the KNN library without creating a direct dependency on hnswlib headers in the public API. Key changes: - Bumped LIB_VERSION from 9 to 10 to reflect API changes - Added `FilterCallback_fn` type alias (`std::function<bool(uint32_t)>`) as a public API for filter callbacks - Added forward declaration for `hnswlib::BaseFilterFunctor` in global namespace to avoid including hnswlib headers in public API - Created `FilterCallbackWrapper_c` class that adapts `FilterCallback_fn` to `hnswlib::BaseFilterFunctor` internally - Updated `KNN_i::CreateIterator()` to accept both: - `::hnswlib::BaseFilterFunctor * pFilter` (for internal use) - `FilterCallback_fn fnFilter` (for public API) - Updated `KNNIndex_i::Search()` to accept both filter types and use the wrapper when `fnFilter` is provided but `pFilter` is null - Updated `CreateIterator()` function to accept and pass both filter types - Changed all `hnswlib::` namespace references to `::hnswlib::` for explicit global namespace qualification Architecture: External code creates a function/lambda implementing the filter logic and passes it as `FilterCallback_fn` to the KNN library. The KNN library then wraps this in `FilterCallbackWrapper_c` which inherits from `hnswlib::BaseFilterFunctor`, allowing it to be passed to hnswlib's search algorithm. This maintains API separation while enabling on-the-fly filtering functionality. This change enables on-the-fly KNN filtering where the search algorithm continues exploring until k filtered candidates are found, rather than finding k total candidates and then filtering them post-search.
sanikolaev
force-pushed
the
issue-4103-knn-filtering
branch
from
7dd0ccf to
bfee605
Compare
5 tasks
Windows test results 5 files 5 suites 18m 6s ⏱️ For more details on these failures, see this check. Results for commit 920f181. |
Linux debug test results 8 files 8 suites 13m 1s ⏱️ Results for commit 920f181. |
Linux release test results 8 files 8 suites 6m 56s ⏱️ For more details on these failures, see this check. Results for commit 920f181. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces a filter callback mechanism that allows external code to pass filter callbacks to the KNN library without creating a direct dependency on hnswlib headers in the public API.
Related PRs:
Key changes:
FilterCallback_fntype alias (std::function<bool(uint32_t)>) as a public API for filter callbackshnswlib::BaseFilterFunctorin global namespace to avoid including hnswlib headers in public APIFilterCallbackWrapper_cclass that adaptsFilterCallback_fntohnswlib::BaseFilterFunctorinternallyKNN_i::CreateIterator()to accept both:::hnswlib::BaseFilterFunctor * pFilter(for internal use)FilterCallback_fn fnFilter(for public API)KNNIndex_i::Search()to accept both filter types and use the wrapper whenfnFilteris provided butpFilteris nullCreateIterator()function to accept and pass both filter typeshnswlib::namespace references to::hnswlib::for explicit global namespace qualificationArchitecture:
External code creates a function/lambda implementing the filter logic and passes it as
FilterCallback_fnto the KNN library. The KNN library then wraps this inFilterCallbackWrapper_cwhich inherits fromhnswlib::BaseFilterFunctor, allowing it to be passed to hnswlib's search algorithm. This maintains API separation while enabling on-the-fly filtering functionality.This change enables on-the-fly KNN filtering where the search algorithm continues exploring until k filtered candidates are found, rather than finding k total candidates and then filtering them post-search.
Notes
This PR includes a temporary change in
cmake/GetHNSW.cmake. It points to an updated version of hnswlib in a branch and will be updated later.