-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Description
Ctrl+C currently does not interrupt execution.
This is an issue when running from an interactive session (i.e. on Casper with qinteractive). In times when the HPC is busy, it can take hours to get a new session, and there's no easy way to kill the plot script without killing the entire session.
This solution below worked for me but I don't know if it has other consequences...
import signal
Add this under worker_init (line 551)
# Let only the parent process handle Ctrl+C and shut down the pool cleanly.
signal.signal(signal.SIGINT, signal.SIG_IGN)
setup_logging(debug=debug)
change block starting in line 593 with this:
# This is needed to avoid some kind of file handle clobbering mumbo-jumbo with netCDF
multiprocessing.set_start_method("spawn")
pool = multiprocessing.Pool(processes=args.procs,
initializer=worker_init,
initargs=(args.debug,))
try:
# Use async map + short waits so KeyboardInterrupt is handled promptly.
result = pool.starmap_async(plotithandler, plotargs, chunksize=1)
while not result.ready():
result.wait(timeout=0.5)
# Surface worker exceptions in parent process.
result.get()
except KeyboardInterrupt:
logger.warning("Ctrl+C received; terminating worker processes")
pool.terminate()
pool.join()
raise SystemExit(130)
else:
pool.close()
pool.join()
logger.info("Done plotting all figures!")
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels