NeMo installation fails due to outdated abandoned dependency on youtokentome

[svandiekendialpad]

Describe the bug

NeMo installation fails with Python >=3.9 because youtokentome does not properly specify it's build dependency on cython. See this issue in their repo.. There is also a PR on that repo to fix that but the author seems to have abandoned the project, its last release was in 2020.

Steps/Code to reproduce bug

Trying to install nemo_toolkit using Python 3.9 or above.

Expected behavior

Installation should succeed.

Environment overview (please complete the following information)

• Environment location: GCP
• Method of NeMo install:poetry add nemo_toolkit[asr]

Environment details

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:

• Debian
• PyTorch 1.13.1 and >2.0
• Python 3.9, 3.10, 3.11

Additional context

This is particularly annoying given that the whole package depends on youtokentome (see requirements_common.txt). So even if users like me only care about the ASR part, youtokentome must be installed.

[titu1994]

Cython is a requirement of NeMo, so that must always be installed before other libraries. However, we have installed NeMo on python 3.9 for some time without issue using pip.

Though the cython itself has been installed a long time ago so that might be an issue.

Our NMT models depend on YTTM, so worst comes to worst we'll have to find an alternative and deprecate these models. But first we'll take a look at possible resolutions

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.

[TechInterMezzo]

Is there a workaround? I can't install nemo_toolkit in my venv under Windows 10.

[LahiLuk]

This issue is very real, because even though there is a workaround when installing individual libraries using pip or poetry, trying to install from requirements.txt or from poetry.lock fails.

The easiest workaround seems to be to merge the PR @svandiekendialpad mentioned in a fork of the original repo, and then install that project before installing nemo-toolkit.

I have taken this approach, creating the fork @https://github.com/LahiLuk/YouTokenToMe. Leaving it here in case it can help someone as a workaround.

[Arno-Riverside]

This is still an issue on my end, Cython is not installed by default on my EC2 servers, and having to preinstall a package for production adds unnecessary complexity.

This is frustrating since the workaround is very straightforward and @LahiLuk already provided the PR.

Since the project is abandoned, another workaround would be to specify git@https://github.com/LahiLuk/YouTokenToMe in the requirements.txt

[svandiekendialpad]

@LahiLuk thanks for your fork. It mentions that it has been tested with Python 3.10. I seem to be able to use poetry add git+https://github.com/LahiLuk/YouTokenToMe.git in Python 3.9 but in 3.10 it fails because pyyaml==5.4.1 is not pep 517 compliant.

[LahiLuk]

Hi @svandiekendialpad, sorry for the late reply. If this is still a problem, could you please open an issue in https://github.com/LahiLuk/YouTokenToMe/issues? Then I can investigate further. Thanks!

[rpomaris]

@titu1994 - can you please take a look at this again? The latest docs say NeMo is dependent on Python 3.10 or higher, but the installation fails with python>3.9. https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/starthere/intro.html#installation

[titu1994]

Fails how? The CI is running Nemo daily on 3.10.12