Testing functionality for uploaded models

[hvgazula]

How do we ensure the integrity of the train/predict/{any other}.py scripts uploaded by the users? Are the scripts really doing their job? Do we test this in trained-models or nobrainer-zoo?

@satra @gaiborjosue

[satra]

through a test via continuous integration in trained-models via the zoo. having a trained model means there should be one example of inference with it, and if the model allows training/finetuning/etc, then examples of that.

[hvgazula]

I don't think a true 'test' is possible but an action to check if the inference ran without errors is possible. Maybe, a 'test' to check if all the expected output files were created constitutes a successful test?

[satra]

i agree that there are different levels of tests. for what constitutes a successful test will require defining a set of benchmarks. but the question here was about integrity of scripts, not performance of the model. i would decouple those concepts. defining a benchmarking suite is part of filling the model card, and assigning which benchmarks are applicable to the model. this doesn't need to run in the CI currently, but should be added as an automation step later on.