Feature: code for Masakhane News and Bloom-lm datasets

[cleong110]

• Create snippets that take in a language code and download from Masakhane News/Bloom-lm
• Combine that data with train.txt, etc. from other datasets
• Make sure it all gets tagged in Weights and Biases!
• Incorporate into repo AND colab notebook for AfricaNLP

[manny-uncharted]

Is this still open, and should the snippets be in a separate file

[cleong110]

@manny-uncharted yes, still open as far as I know. The snippets can be in a separate file, sure. They will eventually be incorporated into one of the notebooks, since that's how we run our experiments for this project.

[manny-uncharted]

Okay thanks I'll get started on this then

[manny-uncharted]

@cleong110 do I make a new branch to make pull requests

[cleong110]

That seems like a good idea to me! The branch by @krypticmouse is essentially our master branch, we should merge that into master, then merge yours

[cleong110]

For now though, just branch off of his https://github.com/krypticmouse/double-bind-training/tree/train-lm-adapter and merge back into his as well.

[manny-uncharted]

@cleong110 apologies for delaying on this task, I've been quite engaged with school work, I've spent some time to study the bloom-lm dataset and the masakhane ner 2.0 dataset.

Here are a few things I noticed.

• The naming conventions differ for both dataset, so I'd have to work around creating a new naming convention, that allows me to capture the different naming conventions.
• also how do I ensure it gets tagged in weights and biases

[krypticmouse]

@manny-uncharted if possible keep the naming conventions consistent with the MAFAND dataset. Also for the weights and bias things add the dataset names you're using as an environment variable then pass them in the tags parameter in the call. Refer to docs in README.

Let me know if you need any help.