Consider forking and maintaining pyctcdecode or switch to torchaudio.models.decoder

[FredHaa]

System Info

transformers[torch-speech]==4.56.2
pyannote-audio==4.0.0

Who can help?

@Rocketknight1

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

With the release of the new pyannote-audio==4.0.0 a few problems arises due to the pyctcdecode dependency which seems to be abandoned.

pyannote-audio==4.0.0 depends on numpy>=2.0.0, but the latest pyctcdecode==0.5.0 (from January 2023) depends on numpy<2.0.0. A PR for numpy 2.0 support has been ignored since February (kensho-technologies/pyctcdecode/pull/116). The restriction of numpy<2.0.0 is arbitrary and was set way before numpy 2.0.0 was announced.

Another problem arises because one of the last commits to pyctcdecode changes the output format of the decoder from tuple to a dataclass, making it incompatible with the current transformers ASR pipeline code..

I.e., currently the only way to use the new pyannote-audio==4.0.0 speaker diarization lib with a Wav2Vec2ProcessorWithLM is by forking pyctcdecode, reverting the main branch to the 0.5.0 state, and then removing the numpy<2.0.0 restriction.

user@host ~> uv pip install "transformers[torch-speech]==4.56.2" pyannote-audio==4.0.0                                                                                                                                                                                                           1
  × No solution found when resolving dependencies:
  ╰─▶ Because transformers[torch-speech]==4.56.2 depends on pyctcdecode>=0.4.0 and pyctcdecode>=0.4.0 depends on numpy>=1.15.0,<2.0.0, we can conclude that transformers[torch-speech]==4.56.2 depends on numpy>=1.15.0,<2.0.0. (1)

      Because pyannote-core==6.0.1 depends on numpy>=2.0 and only pyannote-core<=6.0.1 is available, we can conclude that pyannote-core>=6.0.1 depends on numpy>=2.0.
      And because pyannote-audio==4.0.0 depends on pyannote-core>=6.0.1, we can conclude that pyannote-audio==4.0.0 depends on numpy>=2.0.
      And because we know from (1) that transformers[torch-speech]==4.56.2 depends on numpy>=1.15.0,<2.0.0, we can conclude that pyannote-audio==4.0.0 and transformers[torch-speech]==4.56.2 are incompatible.
      And because you require transformers[torch-speech]==4.56.2 and pyannote-audio==4.0.0, we can conclude that your requirements are unsatisfiable.

Reproduction

uv venv
uv pip install "transformers[torch-speech]==4.56.2" pyannote-audio==4.0.0

Expected behavior

The packages should install

[FredHaa]

Another solution could of course be to convert the Wav2Vec2ProcessorWithLM to use torchaudio.models.decoder instead of pyctcdecode

[Rocketknight1]

cc @eustlb @ebezzam since this is an audio issue - we may need to change or fork some dependencies

[eustlb]

Hey @FredHaa, thanks for raising this issue!
Totally agree we need to move from pyctcdecode to using torchaudio's CTCDecoder. @FredHaa is raising a PR to contribute to something you would be interested in doing? 🤗 Happy to review if so

[FredHaa]

Yes, I can have a look at that 👌

[FredHaa]

Looking at the ctc_decoder builder, it should be possible to construct the decoder using the existing files in the language_model folder in Wav2Vec2ProcessorWithLM repos, by extracting the possible tokens from the unigrams.txt file, and then passing lexicon=unigrams, tokens=extracted_tokens and lm=kenlm_file.

However, the attrs.json file contains options for pyctcdecode which doesn't have a direct correspondence to the torchaudio's CTCDecoder, which means that users might experience a drop in decoding quality once the shift is made.

What is the procedure for this?

[FredHaa]

In order for push_to_hub() to work, the torchaudio CTCDecoder needs to support saving the decoder to a folder. Currently the CTCDecoder is not aware of the path to the original kenlm file, and therefore there is no way to save the decoder so it can be pushed.

I have opened an issue pytorch/audio/issues/4113 and will continue with a PR here as soon as the feature is implemented.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[FredHaa]

Since torchaudio is not responding to my issue (I assume they are busy with porting the library to the stable ABI), I have made a PR to pyctcdecode which adds Numpy 2.0 support to the main branch, while adding support for the old index based interface which transformers relies on:

kensho-technologies/pyctcdecode#117

It likely won't get merged, so anyone having trouble with using pyannote-audio (or any dependency requiring Numpy>=2.0) with transformers[audio] can use my fork:

https://github.com/mediacatch/pyctcdecode

[kho]

Instead of supporting just one decoder implementation, would you please consider setting up a simple Protocol interface so that users can plug in their own decoders of choice? That would be useful for supporting both legacy users of pyctcdecode and new users.

[eustlb]

That's an interesting proposition @kho, we'll look into that

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.