New Response Diversity Tools!

[conconchowchow]

Collaborators: @conconchowchow @Jason-Lee08 @Kathleen0123 (Variegation Team)

also, we'll need to move from [jaredquincy/ember] -> [pyember/ember]

---

Changes

• Created Diversity TestBench (.ipynb notebook) -> to enable testing different forms of diversity scores

  • Added 5 different diversity metrics -> calls ember package to utilize
    • Cosine Similarity
    • Compression Ratio
    • Edit Distance
    • Novelty Score (WIP)
    • Ensembled Score (Cosine + Compression + Edit)

• Added the 5 Evaluators within Ember Repo

  • moved all diversity evaluators own file > diversity_evaluators.py
  • Added usage/examples for Diversity Metrics in this file

• Added a new Operator called DiversityScoringOperator, which takes in EmbeddingProviderModel (from ember.core.registry.model.)

  • in file > diversity_scorer.py

• Upgrading embedding models -> enabling cosine similarity as an evaluator

• Integrated capability models

  • registry/model/providers/provider_capability.py

• Added OpenAIExtendedModel to openai_provider.py file, enabling usage of OpenAI embedding models

  • registry/model/providers/openai/openai_provider.py

• renamed ember/core/utils/logging.py -> ../ember_logging.py to make it distinct from Python's logging module

---

New Dependencies

diversity==0.2.0
spacy==3.8.4
python-Levenshtein==0.27.1
matplotlib==3.10.1

---

Future Considerations / Additions

• Add a diversity-operator example full example for NoN graph (TODO)
• Add Scheduling / Batching to optimize tests for testbench
• Update testbench to call classes instead of within the file (make an example)
• Upgrade embeddings to have a more streamlined way to swap different embedding models (integrating more features from the EmbeddingProviderModel / EnhancedModelRegistry into the main package)