Use pagination

[kingosticks]

When browsing the the stations for a specific country we request all the stations rather than use the API's pagination support. There are (as of today) 9025 US stations returned by Dirble and this data takes ~25 seconds to fetch for me. The default timeout of 5 seconds means this fetch fails. And we then also retry 3 times which is probably why my usage quota has been exceeded.

However, for categories, we don't use all=1 in our request and so we only get the first 20 results. It's not clear how the results are ordered but the Rock category has over 2000 more stations.

[adamcik]

Pagination seems to be max 30 per page, for both categories and countries modes. So even if I just a do a simple _fetch(..., paginate=True) with a while loop to keep trying until we get an empty result we burn 300 queries on the US and 60 or so on Rock.

As for all pagination systems I have to hope that results are stable and sorted by the ID. Which is also why Mopidy will use a pagination token and giving users a next_token if we ever add support.

This also raises a general question of if Mopidy should paginate, having use return 9000 results probably isn't great either...

[adamcik]

Quick (bad) fix, increase the timeout to some high value? :/

[adamcik]

And as things are now, people on the default API plan get ~50 queries a day on average. This might also be a reason to consider having a persistent cache, if we do that with a generous TTL we could burn through the 300 requests for 'US' without as much of a worry.

[kingosticks]

I had to add a 25 second timeout on my work machine to get it working. That's a decent machine on a fast connection. I would imagine a raspberry pi is unusable. I think returning just the first page of results might be reasonable enough.

I don't think it's sorted by id. Also, I did notice their server supports etags so with a cache you'd only have to do each huge request once. Although as I said, once might not even be possible on an rpi.

[adamcik]

Think I have a fix for this mostly done, I'll try and get it out as a PR in case someone wants to double check the fix (if nothing else there are some big shed type things where a second opinion could be nice).