-
Notifications
You must be signed in to change notification settings - Fork 6
Open
Labels
Description
While the jobs are done, EMS happened to be unable to finalize the results and the experiment completion got stuck in 98%. A sudo systemctl restart ems.service recovered the results; But, the output compressed files seams corrupted.
On the Submit Node:
● ems.service - Experiment management service to support GWS
Loaded: loaded (/etc/systemd/system/ems.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2017-07-04 12:09:39 EDT; 6 days ago
Main PID: 281976 (python)
Tasks: 8
Memory: 33.2G
CPU: 6d 8h 32min 30.252s
CGroup: /system.slice/ems.service
├─281976 /usr/bin/python ems.py
├─488301 perl /usr/bin/parallel -P 2 tar jxf ::: Results62.tar.bz2 Results178.tar.bz2 Results181.tar.bz2 Results205.tar.bz2 Results7.tar.bz2 Results20.tar.bz2 Results172.tar.bz2 Results50.tar.b
├─488603 tar jxf Results77.tar.bz2
├─488604 bzip2 -d
├─488607 tar jxf Results98.tar.bz2
└─488608 bzip2 -d
Jul 10 12:53:50 graple-Submit python[281976]: perhaps it is corrupted? *Possible* reason follows.
Jul 10 12:53:50 graple-Submit python[281976]: bzip2: Inappropriate ioctl for device
Jul 10 12:53:50 graple-Submit python[281976]: Input file = (stdin), output file = (stdout)
Jul 10 12:53:50 graple-Submit python[281976]: It is possible that the compressed file(s) have become corrupted.
Jul 10 12:53:50 graple-Submit python[281976]: You can use the -tvv option to test integrity of such files.
Jul 10 12:53:50 graple-Submit python[281976]: You can use the `bzip2recover' program to attempt to recover
Jul 10 12:53:50 graple-Submit python[281976]: data from undamaged sections of corrupted files.
Jul 10 12:53:50 graple-Submit python[281976]: tar: Unexpected EOF in archive
Jul 10 12:53:50 graple-Submit python[281976]: tar: Unexpected EOF in archive
Jul 10 12:53:50 graple-Submit python[281976]: tar: Error is not recoverable: exiting now
On RStudio Console after Attempting to Download the Results:
gzip: results.tar.gz: unexpected end of file
gzip: results.tar.gz: uncompress failed
Sims/Sim78_1/Results/output.nc: Truncated tar archive
tar: Error exit delayed from previous errors.
Warning message:
In untar("results.tar.gz") :
‘/usr/bin/gzip -dc 'results.tar.gz' | /usr/bin/tar -xf '-'’ returned error code 1
Reactions are currently unavailable