Display word frequency under token

[ShanaryS]

This adds a new option TokenFrequencyAnnotation which can be Always, Uncollected only, and Never.

Currently we need an API request to yomitan per token so it's slow and cannot be bulked. To mitigate this, fetching the frequency is non-blocking and will trickle in over time. A full solution will require changes in Yomitan which should hopefully be resolved.

Users will need to have frequency dictionaries installed and they need to be rank-based. Checking for this will also be present the Yomitan PR.

[killergerbah]

What are you planning for the styling? At first glance it feels like using numbers to express the frequency might make the text feel crowded for the average user. But in any case this is pretty awesome work.

[cloudflare-workers-and-pages]

Deploying asbplayer with    Cloudflare Pages

Latest commit:	40a9dc0
Status:	✅  Deploy successful!
Preview URL:	https://ddc6810d.asbplayer.pages.dev
Branch Preview URL:	https://token-frequency.asbplayer.pages.dev

View logs

[ShanaryS]

I changed the in flight loading to use a new class AsyncSemaphore which is from one of my other projects. This maintains the FIFO ordering so that relevant frequencies are calculated first. I also added this.initialized which is set to true on the small initial cache build. This ensures that we only start getting frequencies after the main tokenize work is done (if frequencies aren't supported in tokenize).

What are you planning for the styling? At first glance it feels like using numbers to express the frequency might make the text feel crowded for the average user.

I've only ever thought of using numbers for this. I think using another abstraction like color (something that covers ranges of frequencies) is actually more complex since the user will need to remember what it means along with token status. It would now require a lot more settings too so a user can customize what each level means. Also if we ever add pitch accent we will need to use a token styling for that and so it can become extremely infromation dense.

I think the frequency is useless for everything except uncollected tokens, the only real value of it is to know if something is worth adding to Anki. So if they use uncollected only, they can hover+shift to check the frequencies in Yomitan as needed for everything else to reduce the clutter. This feature only brings value if it can be more convenient than the already convenient Yomitan lookup (aside from future features using frequency for automations).

[killergerbah]

Interesting, expressing frequency with numbers make sense then

[killergerbah]

Debating whether to merge this now or release 1.14 first

[ShanaryS]

I think we should merge and release with 1.14. It's pretty low stakes since it loads it asynchronously and if it's kept as NEVER then it has no effect on the existing features. It also contains a few bug fixes like the Yomitan reset and trackStates not updating properly causing duplicate requests at startup.