Multipart uploads fails on GCS buckets when uniform bucket-level access is enabled

[louisnow]

Error seen

[s3proxy] D 12-16 03:36:42.327 S3Proxy-Jetty-19 o.j.r.i.InvokeHttpMethod:56 |::] >> invoking ObjectAccessControls:get
[s3proxy] D 12-16 03:36:42.327 S3Proxy-Jetty-19 o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Sending request 207806381: GET https://storage.googleapis.com/storage/v1/b/atlan-vcluster-enpla1cp89/o/fc250d3c-bb76-47af-a071-468d190b542c/acl/allUsers HTTP/1.1
[s3proxy] D 12-16 03:36:42.328 S3Proxy-Jetty-19 jclouds.headers:56 |::] >> GET https://storage.googleapis.com/storage/v1/b/atlan-vcluster-enpla1cp89/o/fc250d3c-bb76-47af-a071-468d190b542c/acl/allUsers HTTP/1.1
[s3proxy] D 12-16 03:36:42.328 S3Proxy-Jetty-19 jclouds.headers:56 |::] >> Accept: application/json
[s3proxy] D 12-16 03:36:42.356 S3Proxy-Jetty-19 o.j.h.i.JavaUrlHttpCommandExecutorService:56 |::] Receiving response 207806381: HTTP/1.1 400 Bad Request
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << HTTP/1.1 400 Bad Request
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << Server: UploadServer
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << X-GUploader-UploadID: AHVrFxPu3ewka1oAQY3mzEtt7Cq-j4fbgxInSMMjMXp1N5PbnCYchLc9KX-dySp9xIV0NVVI
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << Vary: Origin
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << Vary: X-Origin
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << Pragma: no-cache
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << Date: Tue, 16 Dec 2025 03:36:42 GMT
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << Cache-Control: no-cache, no-store, max-age=0, must-revalidate
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << Content-Type: application/json; charset=UTF-8
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << Content-Length: 492
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.headers:56 |::] << Expires: Mon Jan 01 00:00:00 UTC 1990
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "{[\n]"
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "  "error": {[\n]"
[s3proxy] D 12-16 03:36:42.357 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "    "code": 400,[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "    "message": "Cannot get legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access",[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "    "errors": [[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "      {[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "        "message": "Cannot get legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access",[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "        "domain": "global",[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "        "reason": "invalid"[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "      }[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "    ][\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "  }[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 jclouds.wire:56 |::] << "}[\n]"
[s3proxy] D 12-16 03:36:42.358 S3Proxy-Jetty-19 o.gaul.s3proxy.S3ProxyHandler:3004 |::] sendSimpleErrorResponse: 400 BadDigest Bad Request {}

Using uniform bucket level access is the default recommended option by google and it looks like it fails

I'm also attaching the whole log

s3proxy.log

We end up with these orphan files in the root of the GCS bucket and the actual file doesn't get created

[gaul]

This will be hard to fix since Apache jclouds is no-longer maintained. Eventually this will be replaced with a new SDK-based storage backend #757 but I have no time to work on this now.

[louisnow]

I can work on this, will it be similar to #940 ?

[gaul]

Exactly the same approach just using Google's SDK. I am willing to merge PRs that lack some functionality, e.g., multi-part upload, access control, but want to make sure that a useful subset of tests from AwsSdkTest pass. Scoping the effort I estimate a day to make something basic, a few days to make it useful, and a week or more to be more complete. The most discouraging part is usually authentication. YMMV.