Added a datatrove based pipeline for filtering tokenized data using scores.#235

Open

BlueCrescent wants to merge 20 commits intomasterfrom

filtering_pipeline

Commits on Oct 27, 2025

Merge branch 'master' into filtering_pipeline
ajude2s
committed

Commits on Oct 29, 2025

refactor: removed reliance on file hashes in the score-based filtering pipeline and adapted the codebase for new changes from main
ajude2s
committed
test: add comprehensive tests for score-based filtering pipeline functionality
ajude2s
committed
chore: remove hardcoded YAML file path from main execution block
ajude2s
committed

Commits on Oct 30, 2025

feat: add Slurm configuration files for filtering pipeline and update execution settings
ajude2s
committed

Commits on Nov 4, 2025

Commits on Dec 9, 2025

Commits on Dec 10, 2025

Commits on Dec 11, 2025