apt_dpkg: retry on HTTP 429 'Too Many Requests'

[bdrung]

Running the system tests on different releases in parallel can cause some tests of test_packaging_apt_dpkg fail:

__________________ test_install_packages_dependencies[deb822] __________________
[...]
apport/packaging_impl/apt_dpkg.py:1326: SystemExit
----------------------------- Captured stderr call -----------------------------
ERROR: Package download error, try again later: Failed to fetch http://ddebs.ubuntu.com/pool/main/p/pcre2/libpcre2-8-0-dbgsym_10.39-3build1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/libs/libselinux/libselinux1-dbgsym_3.3-1build2_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

Retry package download if HTTP 429 'Too Many Requests' errors can be found in the apt.cache.FetchFailedException error. The download is retries with an exponential backoff and giving up after around one hour.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.22%. Comparing base (f713fe1) to head (2aa9439).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #319   +/-   ##
=======================================
  Coverage   84.21%   84.22%           
=======================================
  Files         104      104           
  Lines       20778    20803   +25     
  Branches     3253     3257    +4     
=======================================
+ Hits        17498    17521   +23     
- Misses       2827     2828    +1     
- Partials      453      454    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bdrung]

The retry does not seem to work as intended: https://github.com/bdrung/apport/actions/runs/8718363097/job/23915454500

[bdrung]

We also have failing apt updates:

self = <apport.packaging_impl.apt_dpkg.__AptDpkgPackageInfo object at 0x7f32cca86490>
aptroot = '/tmp/tmpdkc5aafi/cache/Foonux 22.04/apt'
apt_dir = '/tmp/tmpdkc5aafi/config/Foonux 22.04'
fetchProgress = <apt.progress.base.AcquireProgress object at 0x7f32bc36be50>
distro_name = 'ubuntu', release_codename = 'jammy', origins = None
arch = 'amd64'

    def _sandbox_cache(
        self,
        aptroot,
        apt_dir,
        fetchProgress,
        distro_name,
        release_codename,
        origins,
        arch,
    ):  # pylint: disable=too-many-arguments
        """Build apt sandbox and return apt.Cache(rootdir=) (initialized
        lazily).
    
        Clear the package selection on subsequent calls.
        """
        if not self._sandbox_apt_cache or arch != self._sandbox_apt_cache_arch:
            self._clear_apt_cache()
            self._build_apt_sandbox(
                aptroot, apt_dir, distro_name, release_codename, origins
            )
            rootdir = os.path.abspath(aptroot)
            self._sandbox_apt_cache = apt.Cache(rootdir=rootdir)
            self._sandbox_apt_cache_arch = arch
            try:
                # We don't need to update this multiple times.
                self._sandbox_apt_cache.update(fetchProgress)
            except apt.cache.FetchFailedException as error:
>               raise SystemError(str(error)) from error
E               SystemError: W:Download is performed unsandboxed as root as file '/tmp/tmpdkc5aafi/cache/Foonux 22.04/apt/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_jammy_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied), E:Failed to fetch http://ddebs.ubuntu.com/dists/jammy/main/binary-amd64/Packages.gz  429  Too Many Requests [IP: 185.125.190.18 80], E:Failed to fetch http://ddebs.ubuntu.com/dists/jammy-updates/main/binary-amd64/Packages.gz  429  Too Many Requests [IP: 185.125.190.18 80], E:Some index files failed to download. They have been ignored, or old ones used instead.

See https://github.com/bdrung/apport/actions/runs/8718363097/job/23915453469

[schopin-pro]

LGTM. One small nitpick.

[schopin-pro]

nitpick: this is misleading wrt the git log

While it is technically correct because the git log says "around 1h" so if the next delay is 4096s we've already waited 4095s (yay maths), backoff is still a derivation of the wall time and not the actual wall time. I'd really like a comment here to that effect like elapsed time is backoff-1 due to exponentiation so that the next person reading this code doesn't have to come to that conclusion on their own.

[bdrung]

The failing retries:

apport/packaging_impl/apt_dpkg.py:1479: SystemExit
----------------------------- Captured stderr call -----------------------------
WARNING: Package download error, retrying in 1 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 2 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 4 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 8 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 16 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 32 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 64 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 128 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 256 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 512 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 1024 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

WARNING: Package download error, retrying in 2048 second(s): Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

ERROR: Package download error, try again later: Failed to fetch http://ddebs.ubuntu.com/pool/main/c/coreutils/coreutils-dbgsym_8.32-4.1ubuntu1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]
Failed to fetch http://ddebs.ubuntu.com/pool/main/c/curl/libcurl4-dbgsym_7.81.0-1_amd64.ddeb 429  Too Many Requests [IP: 185.125.190.18 80]

So something clearly does not work correctly when retrying. It should succeed after some time.