Semantic Double-Pass Merging for improved keyphrase generation

[qdrddr]

Does it makes sence to utilize SDPM algorythm for keyBERT? This basically looks similarity forward, and then for the second time looks backward. This can be applicable to the keyphrases to better group them and the second pass can remove or split the words in a keyphrase if its not close enough.

Double-pass merging approach first groups content by semantic similarity, then merges similar groups within a skip window, allowing it to connect related content that may not be consecutive in the text. This technique is particularly useful for documents with recurring themes or concepts spread apart.

Here is explanation of how this works.
https://bitpeak.com/chunking-methods-in-rag-methods-comparison/
https://docs.chonkie.ai/chunkers/sdpm-chunker

[MaartenGr]

Hi! Thank you for sharing this. I'm not familiar with this algorithm, but based on the description it does seem like it would be possible to implement something like that here.

The actual implementation in KeyBERT does seem ik would need some significant changes here and there but nothing that shouldn't be possible.

Is this something that you would like to pick up? I currently do not have much time for additional features but perhaps that will change in a couple of months.

[qdrddr]

not sure I can pull this off.
I am a beginner relying on Copilot ))

[qdrddr]

On a related to the optimizations topic,

1. It seems that keyBERT makes sense to implement in conjunction with semantic chunking at the ingestion stage for RAG, so you could reuse those generated vectors by the embeddings model pay less, and enrich metadata with the extracted key phrases. The bottom line is to re-use already generated vectors.
2. Also on the topic of decreasing costs: I think it makes sense when combining keyBERT with keyLLM to feed KeyLLM only with fragments of the original document, where the fragments contain the keywords (instead of feeding the entire document into the LLM to preserve costs). It also seems to be reasonable to incorporate the chunking & injection stage to strike a middle ground in improving quality but decreasing costs. Bottom line is to reduce the number of tokens to LLM.

What do you think? @MaartenGr

[MaartenGr]

It seems that keyBERT makes sense to implement in conjunction with semantic chunking at the ingestion stage for RAG, so you could reuse those generated vectors by the embeddings model pay less, and enrich metadata with the extracted key phrases. The bottom line is to re-use already generated vectors.

That depends on how you want to extract the embeddings. KeyBERT has a function for extracting the embeddings called extract_embeddings which gives back both the document and word embeddings. So any embeddings generated with RAG is unlikely to match those embeddings as they are not the exact same split. Moreover, the way words are chunked is especially important here since KeyBERT assumes a given vocabulary.

Also on the topic of decreasing costs: I think it makes sense when combining keyBERT with keyLLM to feed KeyLLM only with fragments of the original document, where the fragments contain the keywords (instead of feeding the entire document into the LLM to preserve costs). It also seems to be reasonable to incorporate the chunking & injection stage to strike a middle ground in improving quality but decreasing costs. Bottom line is to reduce the number of tokens to LLM.

That would be a nice additional parameter to tweak I think considering each use case is different. For instance, if you have short documents or do not care about the number of tokens, then there is no need.

It would indeed make sense to do this as part of the chunking strategy in a given RAG application but I'm not quite sure what the best approach would be to implement this. It is possible to do this within KeyBERT using the MMR strategy in BERTopic (see here and here, or a variation of that which includes the given keywords. However, I'm not to sure about re-using part of a given RAG pipeline, since there are no standardized components there to use in KeyBERT.