[BLOG] Semantic Search #31
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 1. The text embeddings are output by the inference session. | ||
| 1. These are token-by-token embeddings, so the embeddings for each sequence must be averaged to get an embedding for the entire string. | ||
| 1. Now that `sourceText` and `searchQuery` have been embedded, a search can be performed to find the chunks of text from `sourceText` that most closely match the meaning of the `searchQuery`. This is done by calculating the distance between two embeddings. | ||
| 1. The closest matches are output as semantic matches for the `searchQuery`. |
Contributor
There was a problem hiding this comment.
How do you get back to text from the embeddings?
Contributor
chatasweetie
left a comment
chatasweetie
left a comment
There was a problem hiding this comment.
This came out great and I can't wait to see the images you use to help visual folks
| // Get the length of each of the input sequences by dividing the total length by the count | ||
| int sequenceLength = input.InputIds.Length / count; | ||
|
|
||
| // Create input tensors over the input data using count and sequence length to define the shape |
Contributor
There was a problem hiding this comment.
not sure if tensors is a common known word, if you think so great (I had to look it up and https://www.w3schools.com/ai/ai_tensors.asp is a great resource)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Would love a review of this is any of you have time, still need to put links and images in but content and grammar review would be great.