Adaptation to EAGLE-3

[lekhang4497]

The code for EAGLE-3 has been published. I think the detailed implementation is at https://github.com/SafeAILab/EAGLE/blame/main/eagle/model/cnets.py

Is there any plan to adapt the current code FR-spec EAGLE implementation (in C language) to EAGLE-3?

It would be great if you could give some guidance on how I could adapt the current code base to EAGLE-3.

[Achazwl]

The following repo is modified based on FR-Spec, and supports EAGLE-3, you can refer to it.
By the way, you can create a PR to merge the relevant code into the current repo.

https://github.com/AI9Stars/SpecMQuant/blob/main/llamacu/speculative/eagle3.py
https://github.com/AI9Stars/SpecMQuant/blob/main/src/model/eagle3.cuh

[Achazwl]

Supplement: the key difference is here https://github.com/AI9Stars/SpecMQuant/blob/main/src/model/model.cuh#L169
where we need to store three hidden states for EAGLE-3.

If you have any other questions, feel free to ask me.

[lekhang4497]

Thank you very much for your prompt reply.
Do you have any suggestions on how to debug the C-code part?
Currently, after modifying the code, I have to rebuild the C-code (which takes a very long time) to test the code.

[Achazwl]

You can pip install ninja, which could reduce the rebuild time

[truongdo619]

Hi @Achazwl, I just want to confirm — does it mean that whenever we modify part of a .cuh file (e.g., eagle.cuh), we need to rebuild the entire C++/CUDA codebase and then re-run from . import C in Python to check if the changes work correctly?

Is there a more convenient or faster way to debug the C++/CUDA code without having to rebuild everything each time?

[Achazwl]

Yes, to the best of my knowledge. However, the majority of the compilation time is spent on the flash attention component. With ninja installed, it supports incremental compilation. Given that your changes don't affect the flash attention part, the installation time should theoretically be much shorter—just a few seconds on my machine.

[truongdo619]

Got it! Thank you.