Skip to content

Generalise S3 buckets #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
tonytw1 wants to merge 45 commits into
base: multi-tenant
Choose a base branch
from
Open

Generalise S3 buckets #14

tonytw1 wants to merge 45 commits into from

Conversation

@tonytw1
Copy link
Member

@tonytw1 tonytw1 commented Feb 8, 2025 •
edited
Loading

What does this change?

Expands the concept of an S3 bucket from just the bucket name to a case class fully describing the bucket.
Allows GCP and locally hosted buckets to be used in addition to AWS.

Introduces an S3Bucket class to fully describe a bucket.
The definition of a bucket in configuration moves from bucket name to a triple of name, end point and URL style.

ie.

publishing.image.bucket.name="test-grid-crops"
publishing.image.bucket.endpoint="s3.amazonaws.com"
publishing.image.bucket.pathStyleURLs=false

Decisions about URLs of buckets and keys move to the S3Bucket class which has now enough knowledge to determine this.

How should a reviewer test this change?

How can success be measured?

Who should look at this?

Tested? Documented?

  • locally by committer
  • locally by Guardian reviewer
  • on the Guardian's TEST environment
  • relevant documentation added or amended (if needed)

@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch 6 times, most recently from af00e4e to 831d946 Compare February 15, 2025 12:35
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch 3 times, most recently from f875471 to 23e97d6 Compare February 26, 2025 21:07
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch 2 times, most recently from dfeea9c to 74792d0 Compare March 13, 2025 21:18
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch 9 times, most recently from 01f28c6 to af87d7e Compare April 14, 2025 21:30
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch 8 times, most recently from 12bf2c0 to a9d3863 Compare April 20, 2025 16:56
tonytw1 added 27 commits January 18, 2026 11:12
@tonytw1
Local S3 - host.
92422b8
@tonytw1
Imports.
4f9168c
@tonytw1
Kahuna UI can connect to ingest bucket endpoint.
737a87d
@tonytw1
Log levels.
613234b
@tonytw1
Noisy log.
db121ba
@tonytw1
Noisy log Redact debug.
8cd3a92
@tonytw1
Noisy log.
52fde0c
@tonytw1
GCP S3 interop does not support S3 bulkDelete; reimplement as single …
9bbb334 
…calls.
@tonytw1
Logging tidy up; storeImage
db6f843
@tonytw1
Logging tidy up; thumbnail bucket.
68a2aea
@tonytw1
Clean up; unused unit assignment.
bc0b865
@tonytw1
Logging; lower to debug.
a3cc86c
@tonytw1
Logging; bucket name.
4210b44
@tonytw1
Logging; bucket name.
e81935a
@tonytw1
Inject S3 into Crops from CropperComponents.
b84c145 
Consistent with all other services.
@tonytw1
S3Object takes a full S3Bucket as it's bucket parameter. endpoint par…
03b1291 
…ameter folds into this.
@tonytw1
Incorrect bucket URL.
d6d483e
@tonytw1
S3 objectUrl for bucket and key can move to the bucket (which knows i…
06eeb56 
…t's own URL scheme).
@tonytw1
S3KeyFromURL move to S3Bucket as it know's it's own URL scheme.
49e55bf
@tonytw1
Clean up; further isolating S3Client as the cloudfront concern.
7b0a6c5
@tonytw1
Refactor; extracting the is path style URLs bucket decision.
d5486c3
@tonytw1
S3 path based URLs decision moves to bucket property and config.
d22e9e6
@tonytw1
Reindex process queues project image operations.
9f57424
@tonytw1
Reindex should write projection results as UpsertFromProjectionMessag…
aaeb350 
…e like the controller end point.
@tonytw1
Reindex logging message.
5778440
@tonytw1
Reindex messages are put into the low priority queue.
9e4ffea
@tonytw1
Extract bucket base URL function.
14e5fdf 
Upload bucket connectSource is supplied by the bucket object; knows about path style bucket URLs.

Preserve the objectUrl is http contract.
Not sure if this is important or not.
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch from 6f08a10 to 14e5fdf Compare January 18, 2026 11:16
@tonytw1 tonytw1 changed the title GCP storage buckets Generalise S3 buckets Jan 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tonytw1