Investigate caching issues on ubuntu-20.04 runners.

Error when trying to use pyinstaller: 
```plain
version `GLIBC_2.34' not found (required by /home/runner/.local/venv/bin/python)
```
This error doesn't show up when caching is disabled.

Workflow run: <https://github.com/Qwerty-133/spcache/actions/runs/5363561757/jobs/9731228890>