Avoiding metric fluctuations while training

[saikrishnarallabandi]

Large fluctuations due to less training data.

1. May be better to add additional features that we know will help the task.
2. Regularization ?

[saikrishnarallabandi]

A version of this logic is implemented in the following script:

https://github.com/saikrishnarallabandi/falkon/blob/master/tasks/speech/self_assessed_affect/baseline/local/seqmodels/baseline_attention.py

This has to be stress tested against different tasks and scenarios.

Talk to me :)

[saikrishnarallabandi]

Some resources:

https://datascience.stackexchange.com/questions/31113/validation-showing-huge-fluctuations-what-could-be-the-cause

http://cs231n.stanford.edu/reports/2017/pdfs/300.pdf

https://www.quora.com/What-might-be-the-cause-of-the-many-fluctuations-in-validation-accuracy-during-training-of-my-deep-neural-network

https://www.datacamp.com/community/tutorials/towards-preventing-overfitting-regularization

[saikrishnarallabandi]

Does this fluctuation have something to do with CDF?

Figure 1 in https://arxiv.org/pdf/1803.00047.pdf

[saikrishnarallabandi]

@peterwu99

[saikrishnarallabandi]

Pretraining on Google level data might help (?) @peterwu99