diff --git a/README.md b/README.md index ed84bc6..10080f9 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Helpful scripts for working at NCI To use these scripts on NCI infrastructure do the following: ``` -module use /g/data/hh5/public/modules +module use /g/data/xp65/public/modules module load nci-scripts ``` @@ -22,7 +22,7 @@ $ uqstat qcost ------ -Calculates what it would cost, in SU, for a job submitted to the PBS system with the same configuration. +Calculates what it would cost, in SU, for a job submitted to the PBS system with the same configuration. The information is all contained in the [PBS queue information is provided by NCI](https://opus.nci.org.au/display/Help/Queue+Limits) but it can be tedious to determine which configuration of queue and memory request should be used to minimise job cost. `qcost` was created to make this process easier. @@ -32,8 +32,8 @@ Usage: ./qcost -h usage: qcost [-h] -q QUEUE -n NCPUS -m MEM [-t TIME] -Return what it would cost (in SUs) for a PBS job submitted on gadi with the same -configuration. No checking is done to ensure requests are within queue limits. +Return what it would cost (in SUs) for a PBS job submitted on gadi with the same +configuration. No checking is done to ensure requests are within queue limits. optional arguments: @@ -59,7 +59,7 @@ $ qcost -q normal -n 4 -m 60GB -t 3:00:00 gadi_jupyter ------------ -Run a Jupyter notebook on Gadi, displaying it in a local web browser. +Run a Jupyter notebook on Gadi, displaying it in a local web browser. **THIS MUST BE RUN FROM YOUR OWN COMPUTER** @@ -67,7 +67,7 @@ So the first step is to clone this repository to your local workstation. It works on mac/linux/windows (for windows using e.g. [git bash](https://gitforwindows.org/) or [Windows Ssubsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install)). -By default the script will spawn a job with a single CPU, 4GB of memory +By default the script will spawn a job with a single CPU, 4GB of memory and a walltime of 1 hour. Command line options can be used to alter the resources requested, the @@ -106,13 +106,13 @@ Queue Options: ``` -You will need to have ssh keys setup for logging into gadi. There is a +You will need to have ssh keys setup for logging into gadi. There is a [guide on how to do this on the CMS Wiki](http://climate-cms.wikis.unsw.edu.au/CLEx_induction#Step_2:_Set_up_your_Connection). vdi_jupyter.py -------------- -Run a Jupyter notebook on +Run a Jupyter notebook on [NCI Virtual Desktop Infrastructure (VDI)](https://opus.nci.org.au/display/Help/VDI+User+Guide), displaying it in a local web browser (run from your own computer). To run: @@ -120,11 +120,11 @@ To run: python nci_scripts/vdi_jupyter.py ``` -This script requires the `pexpect` package. To make a `conda` environment with the required +This script requires the `pexpect` package. To make a `conda` environment with the required packages: ``` conda env create -f nci_scripts/vdi_environment.yml -``` +``` then activate the environment: ``` conda activate jupyter_vdi diff --git a/gadi_jupyter b/gadi_jupyter index f9ce8f3..d42e7a4 100755 --- a/gadi_jupyter +++ b/gadi_jupyter @@ -177,9 +177,16 @@ set -eu WORKDIR="$WORKDIR" mkdir -p "\$WORKDIR" -# Check if a hh5 member -if [ ! -d /g/data/hh5 ]; then - echo "ERROR: \$USER is not a member of hh5. Join at https://my.nci.org.au/mancini/project/hh5/join" >&2 +# Check if an xp65 member +if [ ! -d /g/data/xp65 ]; then + echo "ERROR: \$USER is not a member of xp65. Join at https://my.nci.org.au/mancini/project/xp65/join" >&2 + echo "x x x x ERROR" + exit +fi + +# Check if an ik11 member +if [ ! -d /g/data/ik11 ]; then + echo "ERROR: \$USER is not a member of ik11. Join at https://my.nci.org.au/mancini/project/ik11/join" >&2 echo "x x x x ERROR" exit fi @@ -200,8 +207,8 @@ cat > "\$WORKDIR/runjp.sh" < # diff --git a/qsubs b/qsubs index bd7deb5..100a5bc 100755 --- a/qsubs +++ b/qsubs @@ -2,7 +2,7 @@ set -eu -storage="gdata/hh5" +storage="gdata/xp65" for p in $(id -nG); do if [ $p = "access.dev" ]; then continue; fi if [ $p = "access.admin" ]; then continue; fi diff --git a/qtools.py b/qtools.py index 43097be..7504786 100644 --- a/qtools.py +++ b/qtools.py @@ -1,4 +1,4 @@ -#!/g/data/hh5/public/apps/nci_scripts/python-analysis3 +#!/g/data/xp65/public/apps/nci_scripts/python-analysis3 # Copyright 2020 Scott Wales # author: Scott Wales # diff --git a/uqstat b/uqstat index da1392d..0d450b0 100755 --- a/uqstat +++ b/uqstat @@ -1,4 +1,4 @@ -#!/g/data3/hh5/public/apps/nci_scripts/python-analysis3 +#!/g/data/xp65/public/apps/nci_scripts/python-analysis3 import pandas import subprocess @@ -80,7 +80,7 @@ def nqstat_df(project): token = pymunge.encode().decode('utf-8') headers = { 'Authorization': "MUNGE %s" % (token) } params = {'project': project} - + response = requests.get(url, params=params, headers=headers, timeout=120.0) response.raise_for_status() diff --git a/vdi_jupyter.py b/vdi_jupyter.py index a0ef572..f84ee1a 100755 --- a/vdi_jupyter.py +++ b/vdi_jupyter.py @@ -208,7 +208,7 @@ def main(args): logging.info('Unable to launch new VDI session:\n'+r.before.decode()) params.update(m.groupdict()) - time.sleep(2) # TODO: instead of waiting, should check for confirmation + time.sleep(2) # TODO: instead of waiting, should check for confirmation # use has-started logging.info("Determine jobid for VDI session...{jobid}".format(**params)) @@ -222,7 +222,7 @@ def main(args): logging.info("Running Jupyter on VDI...") setupconda = params.get('setupconda', - """module use /g/data3/hh5/public/modules + """module use /g/data/xp65/public/modules && module load conda/analysis3-unstable """.replace('\n', ' ')) @@ -257,7 +257,7 @@ def main(args): # optional: terminate to close the vdi session? def main_argv(): - + args = parse_args(sys.argv[1:]) main(args)