Suggestions & New Features

[pfliu-nlp]

• we can either remove the restriction of version number or relax the restriction by using >= instead of =
  https://github.com/ExpressAI/omneval/blob/master/requirements.txt#L1

• Can we replace the huggingface datasets with datalab?

• Given a dataset (from DataLab) and PLM , can we implement an interface that can generate the predictions, specifically,

  • Input: dataset from datalab, PLM
  • Output: predictions:List over all test samples

[pfliu-nlp]

• what if users want to test a dataset that current omneval doesn't support? I guess they need to write a config like this, could we make this process relatively more automatic? (for example, dynamically generating a dataset config based on the dataset name they provided)

[pfliu-nlp]

• what if users want to evaluate a new PLM, do we have any constrains on it? For example, what types of PLMs do we support now?