pip install ewatercycle does not work nicely with Snellius

[MarkMelotto]

When installing ewatercycle on a Snellius machine, you get the following error:

ESMValToolNotFoundError: ESMValTool or ESMValCore packages have not been found.

Likely because ewatercycle dependencies are not installed
in a conda environment.

Please install ewatercycle with:

curl -o conda-lock.yml https://raw.githubusercontent.com/eWaterCycle/ewatercycle/main/conda-lock.yml
conda install mamba conda-lock -n base -c conda-forge -y
conda-lock install --no-dev -n ewatercycle
conda activate ewatercycle
pip install ewatercycle

---

Snellius does not have conda installed by default, so I am looking to see if there is another way

[BSchilperoort]

Mamba can be used as well. You have to load the right modules for this

see for example;
https://servicedesk.surf.nl/wiki/spaces/WIKI/pages/17826008/User+Manual+-+Teacher#UserManualTeacher-MakingsuretheCondaenvironmentgetsusedtostarttheJupyterServer

For more info on modules; https://servicedesk.surf.nl/wiki/spaces/WIKI/pages/30660245/Environment+Modules

[BSchilperoort]

Miniconda might also be available; https://servicedesk.surf.nl/wiki/spaces/WIKI/pages/30660252/Jupyter+Notebooks+on+Snellius#JupyterNotebooksonSnellius-UsingCondavirtualenvironmentswithSnelliusJupyterHub

[MarkMelotto]

@BSchilperoort I noticed that this is just esmvaltool that needs different requirements that are not on PyPi.
I am trying to use miniconda, but that will also throw errors due to system rights, but I am still working on it!

[BSchilperoort]

module load 2025
module load Mamba/24.9.0-0
mamba init
source ~/.bashrc
mamba create -n lock conda-lock -c conda-forge
mamba activate lock
conda-lock install --no-dev -n ewatercycle
mamba activate ewatercycle
pip install ewatercycle

should work. works on snellius 😉

[BSchilperoort]

Note that there were docs for this already; https://ewatercycle.readthedocs.io/en/latest/infrastructures.html#migrate-from-hpc-to-cluster-snellius-guide
If anything these is incorrect/can be described better it would be good to do it now you're busy with it.

[MarkMelotto]

@BSchilperoort thanks a lot!
I guess that does work on the login nodes, because that would be the default workflow (of course :) )
For the onDemand jupyterHub it does not work.

This where I want to test, but I think I will skip that part for now...

[BSchilperoort]

When you have done mamba init it registers mamba in your profile in .bashrc (it'll be automatically available without doing module load again).

The envs are also in your home dir;

> mamba env list
...
ewatercycle              /home/username/.conda/envs/ewatercycle

This means you can register these as a jupyter kernel, and use them on OnDemand as well.

[BSchilperoort]

See this step on the docs; https://servicedesk.surf.nl/wiki/spaces/WIKI/pages/30660252/Jupyter+Notebooks+on+Snellius#JupyterNotebooksonSnellius-UsingPythonvirtualenvironmentswithSnellius'JupyterHub

# Install the virtual environment as custom kernel. It will show up in the Jupyter Notebook Server with the name passed to the '--name' argument.

[MarkMelotto]

@BSchilperoort it must be me, but it just does not work for me.
Also now I am trying to do it on DelftBlue, together with issue #483 .

[BSchilperoort]

it must be me, but it just does not work for me.

well, snellius is down today and tomorrow haha.

What part isn't working? Registering the kernel?

[MarkMelotto]

yep but also yesterday it did not work.
I can get a different kernel working with:
python -m ipykernel install --user --name=ewatercycle

then it shows up, but I will always run into the same error where ewatercycle isnt installed via the conda-lock?
I also tried various combinations of the guide of the surf service desk.

I feel like this should easily work, but I am missing something.

[BSchilperoort]

If you followed this guide and it still doesn't work, you can also just open a ticket at the surf service desk to ask for help.

[MarkMelotto]

I will do that when Snellius is up again tomorrow!
The weird thing is that esmvaltool is successfully installed in the correct environment, yet it still does not work.

[BSchilperoort]

The weird thing is that esmvaltool is successfully installed in the correct environment, yet it still does not work.

That's weird because when I ran the following I could just do "import ewatercycle" and it ran fine;

module load 2025
module load Mamba/24.9.0-0
mamba init
source ~/.bashrc
mamba create -n lock conda-lock -c conda-forge
mamba activate lock
conda-lock install --no-dev -n ewatercycle
mamba activate ewatercycle
pip install ewatercycle

should work. works on snellius 😉

Which python code failed?

[MarkMelotto]

ah to add to my earlier comment: I am on DB.
Ah yes the import ewatercycle runs fine!
Importing ewatercycle.forcing is causing the issue.

I am starting to think that there is a bug there...