Allow outdated files on mirrors #188
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Feb 14, 2025
Closed
elboulangero
changed the title
Collaborator
|
Looks good. Needs rebase. As long as this is off by default, this works. |
I find it easier to break long lines and have more variables with meaningful names. Maybe just a matter of taste.
In preparation for the next commit
The new setting AllowOutdatedFiles allows user to define which files are
allowed to be outdated on the mirrors, and for how long.
The user defines a list of rules, each rule is of the form:
- Prefix: matched against the beginning of the path of the requested file
- Minute: if Prefix matches, how long the file is allowed to be oudated
AllowOutdatedFiles is a list of rules, they are checked in order, and
the first rule that matches is selected.
Note that, when a rule matches, the filesize check is also disabled for
this file. As it wouldn't make much sense if we allowed a file to be
outdated, but didn't allow it to be of a different size.
Now, here's the use-case for this setting.
For a Debian-like distribution, the directory `/dists` (aka. the
metadata of the repository) contains a lot of files that are updated
in-place. Each time the repository is updated, and immediately after
mirrorbits rescans the local repo, mirrorbits redirects all the traffic
for those files to the fallback mirror, since they have a new modtime, a
new size, and mirrorbits doesn't know yet any mirror with those new
files. It's only after 1) mirrors sync with the origin repository and 2)
mirrorbits scans the updated mirrors, that it can redirect traffic to
mirrors again.
For more details in a real-life setup: Kali Linux is a rolling distro,
the repository is updated every 6 hours, and mirrors are scanned every
hour. In effect, it means that every 6 hours, mirrorbits redirects most
of the metadata traffic to the fallback mirrors, then it takes around 1
to 2 hours before all the mirrors are scanned and traffic flows back to
normal. Then again, 4 times a day.
To prevent that, Kali uses the following setting:
```
AllowOutdatedFiles:
- Prefix: /dists/
Minutes: 540
```
Cf. etix#85 for more details.
Add TestMain to make sure that the configuration is always loaded, as the Filter function needs to get some configuration values. Add variables noFileInfo and noClientInfo variables to improve readability.
elboulangero
force-pushed
the
github/allow-outdated-files
branch
from
6faafbf to
7e4e91b
Compare
Contributor
Author
|
Rebased.
Yep, off by default of course. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR applies after #186, therefore it will need to be rebased after/if #186 gets merged. Only the 6 last commits really belong to this PR.
No need to review it before #186 gets sorted out.
This PR replaces #147
It fixes #85 for Debian-like distributions. The setting proposed here is kind of minimal, but it should be easy to extend it in the future (ie. match files with regex rather than prefix), if ever a needs arise.
The new setting
AllowOutdatedFilesallows user to define which files are allowed to be outdated on the mirrors, and for how long.The user defines a list of rules, each rule is of the form:
Prefix: matched against the beginning of the path of the requested fileMinute: if Prefix matches, how long the file is allowed to be oudatedAllowOutdatedFilesis a list of rules, they are checked in order, and the first rule that matches is selected.Note that, when a rule matches, the filesize check is also disabled for this file. As it wouldn't make much sense if we allowed a file to be outdated, but didn't allow it to be of a different size.
Now, here's the use-case for this setting.
For a Debian-like distribution, the directory
/dists(aka. the metadata of the repository) contains a lot of files that are updated in-place. Each time the repository is updated, and immediately after mirrorbits rescans the local repo, mirrorbits redirects all the traffic for those files to the fallback mirror, since they have a new modtime, a new size, and mirrorbits doesn't know yet any mirror with those new files. It's only after 1) mirrors sync with the origin repository and 2) mirrorbits scans the updated mirrors, that it can redirect traffic to mirrors again.For more details in a real-life setup: Kali Linux is a rolling distro, the repository is updated every 6 hours, and mirrors are scanned every hour. In effect, it means that every 6 hours, mirrorbits redirects most of the metadata traffic to the fallback mirrors, then it takes around 1 to 2 hours before all the mirrors are scanned and traffic flows back to normal. Then again, 4 times a day.
To prevent that, Kali uses the following setting: