Download stuck for long running jobs

[ani-ghosh]

Sometimes file downloads getting stuck in midway. Need ctrl+c to escape and move to next download. One example situation shown below.

Possible solutions and new feature request:

1. Add a time-out in the download function
2. Compare size of downloaded file and content-length from the httr::post request and attempt to re-download if the numbers don't match
3. Use another alternative to the progress bar showing download status

|======================================================================| 100%
|================== | 25%^C
Warning messages:
1: In cmr_download(urls = fileurls, path = path, username = cred$user, :
failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2004.06.17/MYD13Q1.A2004169.h18v07.006.2015154155539.hdf
2: In cmr_download(urls = fileurls, path = path, username = cred$user, :
failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2009.10.08/MYD13Q1.A2009281.h18v07.006.2015195132705.hdf
3: In cmr_download(urls = fileurls, path = path, username = cred$user, :
failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2009.10.08/MYD13Q1.A2009281.h18v08.006.2015195135818.hdf
4: In cmr_download(urls = fileurls, path = path, username = cred$user, :
failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2009.10.08/MYD13Q1.A2009281.h17v07.006.2015195135725.hdf
5: In cmr_download(urls = fileurls, path = path, username = cred$user, :
failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2009.10.08/MYD13Q1.A2009281.h17v08.006.2015195135815.hdf
|======================================================================| 100%
|======================================================================| 100%
|============ | 18%^C
Warning message:
In cmr_download(urls = fileurls, path = path, username = cred$user, :
failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2013.05.17/MYD13Q1.A2013137.h17v07.006.2015260174324.hdf

|======================================================================| 100%

[wildintellect]

• We should look at timeout options for the download, which may depend on download backend (e.g. we might need to specifically force curl).
• We can get the expected file size from the header 'Content-Length' and compare to file.size(), there may also be a check method in the download backends.
• If the Data Source provides a hash, then we could compare hash values.

Note: Some backends actually support resume, so maybe on timeout we should attempt to resume the download of a particular file.