Just update #1
Open
Just update #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
openstack-mirroring
force-pushed
the
master
branch
from
3f3f3c4 to
479a010
Compare
openstack-mirroring
force-pushed
the
master
branch
from
dd19613 to
aa04c3e
Compare
openSUSE stopped providing OpenStack packages some time ago. Co-authored-by: Takashi Kajinami <kajinamit@oss.nttdata.com> Change-Id: I60fcbac1cb6ec370de565903375be411e8fc3415
The timing stats decorators are moved from utils to base_storage_server.py to avoid a circular import of HTTPException. Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Change-Id: Idc4b52de0a04ebfc0e353162bd791d4e0e20eac3
Swift already supports that much, at least. AWS used to not support any conditional PUTs, but that's changed somewhat recently; see - https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/ - https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/ Drive-By: Fix retry of a CompleteMultipartUpload with changed parts; it should 404 rather than succeed in writing the new manifest. Change-Id: I2e57dacb342b5758f16b502bb91372a2443d0182
When a DiskFile is opened, its on-disk files are first listed and then
xattr metadata is read. If a listed file no longer exists when its
metadata is read then a DiskFileNotExist exception was raised which
would previously cause the object server to return a 404.
It isn't appropriate to return 404 when the list of on-disk files
suggested that the object existed (i.e. included a .data file) and
there is no evidence that the object no longer exists: the missing file
may well have been *replaced* by a concurrent request.
For example, if a POST request races with another concurrent POST
request, the object's .meta file may be replaced by a new .meta file,
and reading the original .meta file will fail. Similarly, if a POST
races with a concurrent PUT request, the PUT request may replace the
.data file causing the POST request handler to fail to read the
original .data file. In neither case has the object ever been
non-existent.
This issue was observed during a period of many concurrent POST
requests to an object in a production system. A significant number of
the POSTs returned 404 despite the object never being deleted.
This patch modifies DiskFile to raise a new DiskFileStateChanged
exception when metadata cannot be read from a data, meta, or ts file
that was listed in the object's datadir. The object server may translate
this exception to a 503 response rather than a 404, depending on the
request method:
* POSTs, GETs and HEADs that encounter this transient loss of a
file will now return 503 rather than 404. The DiskFile may
still exist.
* DELETEs that encounter this transient loss of a file would
previously have proceeded but will now return 503: a replacement
file may contain an X-Delete-At that would have prevented the
DELETE proceeding.
* PUTs that encounter this transient loss of a file will continue to
proceed as before: a replacement meta file may contain an
X-Delete-At that requires updates to the expirer queue but those
updates would presumably be handled by the concurrent POST that
has replaced the meta file; a replacement .data file will remain
on disk until the DiskFile is next opened and one of the .data
files will be cleaned up.
Closes-Bug: #2054791
Co-Authored-By: Tim Burke <tim.burke@gmail.com>
Change-Id: I2f698c25ed65b236e851e5a307d48a12cef62b33
Modified code to use native labeled metrics API introduced by the Related-Change. Co-Authored-By: Tim Burke <tim.burke@gmail.com> Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Co-Authored-By: Shreeya Deshpande <shreeyad@nvidia.com> Co-Authored-By: Clay Gerrard <clay.gerrard@gmail.com> Related-Change: I115ffb1dc601652a979895d7944e011b951a91c1 Change-Id: I5a96626862644fe23a0d4bc53e85989770c8ceec
Change-Id: Id5583e3a1e4515ec3c8a972f647aaaabfba673bc Related-Change: I2e57dacb342b5758f16b502bb91372a2443d0182
…uests Change-Id: I05336b700120ab5fcf922590d6a12f73112edb50
Locally, this reduced the test time from 240s to 0.16s. Change-Id: I5a4786b6782c06f8e6bd9fab5d4dae683a970242
Change-Id: I0a27efe897b4e8ce2c21da1a3603a2a77c02eb69
Previously (at least on OSX) this could lead to errors like socket.gaierror: [Errno 8] nodename nor servname provided, or not known when we actually went and tried to connect to send the update (!!) Change-Id: I86f6e731d1ee273c6772974ce597ac91be3937be
We're increasingly trying to move toward having S3 tests that pass against both AWS and Swift. Especially as we move to replace our old boto(2) tests, it would be nice to have the replacements be cross- compatible from the get-go. However, there was previously a risk in leaning too-heavily on that test suite: only the functional test suite would be exercised in the gate with Keystone. Now, run both test suites to ensure we don't leave implementation gaps in s3token. Change-Id: I873c32107b4174bd2968097b5fe96d97f260a256
…g requests The labeled timing metrics previously have sample_rate similar to those added for non-labeled metrics. However the down sampling has not been helpful when using labeled metrics to investigate customer issues, for example those related to object server REPLICATE requests. This patch changes labeled timing metrics to not have sample_rate. The non-labeled metrics are unrelated to this effort thus not changed. Related-Change: I05336b700120ab5fcf922590d6a12f73112edb50 Change-Id: Ia6e856ffaf8fd1b4a905e6976ebdc62ed5ddf32f
…ebugging requests"
Swift does not return all the parameters of objects in a listing (e.g. ChecksumType and ChecksumAlgorithm) so pop these from listings before making assertions. Change-Id: Ieb7a9783731c11f1c08db398eae07ffafa127460
Change-Id: Iea1adfb93891e4bde62a11bfcef478d9b9696fd4
Change-Id: Ic0e49ceed5afff093952da5a282c043a51d7c490 Signed-off-by: Jianjian Huo <jhuo@nvidia.com>
Use an instance of Timestamp.zero() as the default timestamp when X-Timestamp is not found in object metadata. Change-Id: I68b6977312136fe82fbf0404badfe37818978564 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Replace hard-coded timestamps/versions with dynamically generated timestamps/versions in unit tests. Change-Id: I0b36078c004f20f0cda4c91de0da17e382766fc4 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Use Timestamp.now() rather than time.time() when building a version name for a delete marker. The _build_versions_object_name method already converts the float time to a Timestamp, so this patch makes no behavioural change but enables tests to uniforrmly mock Timestamp.now rather than time. Replace hard-coded timestamps/versions with dynamically generated timestamps/versions in unit tests. Add a test where the response to a GET for the current object has x-timestamp and x-backend-timestamp headers. This verifies that x-timestamp is preferred over the last-modified header when version name is created. Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Change-Id: Ia07ebac62129a3456e2bdafc0d086db5a2788a35 Signed-off-by: Matthew Oliver <matt@oliver.net.au> Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: Ic989c84effb6aad72a945dab6e2d4421e6bf5600
Relocate some timestamp related tests from test_utils.py to test_timestamp.py. The tested functions were relocated to timestamp.py in the Related-Change. Related-Change:I5b7ab3b2c150ec1513b3e6ebc4b27808d5df042c Change-Id: I87f8f4e62b108f9e50510509ffd0ea3b3fb03177 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Add swob.parse_date_header() to encapsulate conversion of a Date header to seconds. Use the new helper to replace duplicate code in versioned_writes middlewares. Ensure that calls to _build_versions_object_name in these middlewares uniformly pass a string representation of the object version timestamp. Drive-by: remove unnecessary mock of time from a unit test. Change-Id: I570805f90c89fc8f5f00f35ae1c12de0e4ce7b22 Signed-off-by: Alistair Coles <alistairncoles@gmail.com> Signed-off-by: Jianjian Huo <jhuo@nvidia.com> Co-authored-by: Jianjian Huo <jhuo@nvidia.com>
Caracal has gone EOM; add Flamingo job. Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: Ic50ee837492e84750b4ca6edd97595c761725c42
Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: I4bdd308b5473b27cc64e6e3629e50fe9e8a6217f
Object GET response x-timestamp headers should have the normal format, not the internal format. Change-Id: I254bb7e9d3a1705716d13957872dfe8e6aeec6f4 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Add assertions to proxy unit tests to verify that when the proxy adds an x-timestamp header to PUT, POST and DELETE requests then the value is valid. Add assertions to s3api unit tests to verify that the s3api adds a valid x-timestamp header to some swift requests: - object PUT requests - multi-upload part PUT requests Note: The addition of these assertions is not intended to affirm that it is necessary or appropriate for s3api to add X-Timestamp to these requests (which is questionable). The tests are added to verify that when the header is added then the value is *valid*. Add assertions to s3api unit tests to verify that the s3api does not add an x-timestamp header to other swift requests: - multi-upload complete requests - multi-delete requests - bucket PUT and DELETE requests Note: It is left to other middleware or the proxy server app to add the x-timestamp header to these requests, which is probably quite appropriate. The mocked_http_conn helper is preferred over set_http_connect in tests because it captures backend requests and detects unused responses. Change-Id: Iab68f4b596dfa7f23bc587fbde859d854faff87f Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
test/probe/test_container_failures.py: An x-backend-put-timestamp header has the internal string format so the string representation will not necessarily cast to a float; cast an x-backend-put-timestamp header to a Timestamp first and then cast to float. test/probe/test_object_expirer.py: In a response, an x-backend-timestamp header has the internal string format whereas an x-timestamp header has the normal format, so the two will not necessarily be equal. Use x-backend-timestamp consistently to compare object create_timestamp. Change-Id: I89ed840e05a5f71388d3bdb596f38f62f901b0bb Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Define the sample timestamp as instances of Timestamp rather than their internal format string representation. This enables following changes to manipulate the timestamps before they are serialized to strings. Change-Id: I2152f55fa0c38650ed3c7431e65c68d0255038f3 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: I2d1c0443dc1685e3133a8f720181fc0ae7b11c19
Currently the db_replicator will clean up files older then reclaim_age in the device_tmp directory, but it doesn't clean up any old .tmp files located in the db_dir directory. These .tmp files are created in initialize broker method, which will also then do a rename to the final db file name. We've found a bunch in production along side the db files themselves so there must be an edgecase where it's not renamed, maybe a race condition with replication or the PUT. In anycase, we now want to move the sharding freshdbs to be initially created in the same manner, <db_dir>/fresh<uuid4>.db.tmp and then rename once all configured. So there is a chance that these might be left around from time to time. So would be a good idea to clean them up. This patch adds a new `_reclaim_tmp_dbs` method that is called from `_reclaim` that will generate a list of *.tmp files which is then passed to `unlink_paths_older_than`, cleaning up .tmp files that are older then reclaim_age. Change-Id: If30005269d40a1a3f711008b5d3b863efc9fb683 Signed-off-by: Matthew Oliver <matt@oliver.net.au>
Add api label to determine if the original request is from S3 client or swift. labels['api'] = 'S3' or 'swift' Change-Id: I26f75fb61ba0d13ca16810f13cc6cc193898211a Signed-off-by: Yan Xiao <yanxiao@nvidia.com> Signed-off-by: Shreeya Deshpande <shreeyad@nvidia.com>
Previously, test_reclaim_tmp_files assumed a particular order to the
result of os.listdir. However, no order is guaranteed from python
os.listdir and the test would fail on some platforms (including
macos).
[1] "The list is in arbitrary order"
https://docs.python.org/3/library/os.html#os.listdir
Change-Id: If26c961c2133ec48c32a29e37a4a427aa8a4c818
Related-Change: If30005269d40a1a3f711008b5d3b863efc9fb683
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
The py3 tag is preserved with multiple tags in the swift-*-image jobs. Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: Ic1a5f5ed4ab5fe0fd278083a919efabf39f72c56
The author has observed this test fail because the request somehow succeeded in connecting to 10.0.0.x addresses, meaning that node timings *were* unexpectedly set. Even when the test behaves 'normally' there is a >= 0.5 sec delay waiting for a connection timeout. This is unnecessary: we can simply mock the connections to raise Timeouts immediately. Change-Id: Iff2f0e6dffca73eb689bf354e278e0f727f881af Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: I53445f574953d9d47dd93d6819060e4b3cabee75
Signed-off-by: Matthew Oliver <matt@oliver.net.au> Change-Id: Ieaf9ca9a67e40a7133f56f4fde86a88ba4666ac4
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.