Skip to content

Libvips image loading #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
tonytw1 wants to merge 56 commits into
base: gcp-storage-buckets
Choose a base branch
from
Draft

Libvips image loading #15

tonytw1 wants to merge 56 commits into from

Conversation

@tonytw1
Copy link
Member

@tonytw1 tonytw1 commented Jul 27, 2025 •
edited
Loading

What does this change?

Image ingestion and thumbnail generation implemented using libvips.
Around a 4x reduction in the time taken to ingest an image.

The getColorModelInformation call to ImageMagick identify was one of the unexpectedly expensive (seconds for large image) calls.
We are avoiding this by inferring colourModel from VIPS metadata rather than the output of identify.

Image dimension and orientation metadata is sourced from libvips.

Uses vips-ffm as the vips Java binding.

Opinionated decision to stop generating the optimisedPng pre renders (because vips proxy is faster enough for all image types).
Kahuna ignores existing optimisedPng files.

Tests will need to know the location of libvips libraries when run on an Apple Silicon brew libvips install.

sbt -Dvipsffm.libpath.vips.override=/opt/homebrew/lib/libvips.dylib -Dvipsffm.libpath.glib.override=/opt/homebrew/lib/libglib-2.0.0.dylib -Dvipsffm.libpath.gobject.override=/opt/homebrew/lib/libgobject-2.0.0.dylib test

How should a reviewer test this change?

How can success be measured?

Who should look at this?

Tested? Documented?

  • locally by committer
  • locally by Guardian reviewer
  • on the Guardian's TEST environment
  • relevant documentation added or amended (if needed)

@tonytw1 tonytw1 changed the title Libvips Libvips image loading Jul 27, 2025
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch from 33a2a52 to f49a509 Compare August 2, 2025 18:23
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch from f49a509 to a3cfe7a Compare August 2, 2025 19:38
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch from a3cfe7a to f23d157 Compare August 3, 2025 10:41
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch 7 times, most recently from 58edbd5 to bd3a973 Compare August 9, 2025 17:44
@tonytw1 tonytw1 force-pushed the libvips branch 2 times, most recently from c8f8217 to c96ff82 Compare August 11, 2025 19:06
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch 3 times, most recently from a232cac to 0d3403d Compare September 21, 2025 18:42
@tonytw1 tonytw1 force-pushed the libvips branch 5 times, most recently from d296629 to 25c0148 Compare September 28, 2025 10:12
@tonytw1 tonytw1 force-pushed the gcp-storage-buckets branch 5 times, most recently from b8041b2 to 2e351c4 Compare November 23, 2025 12:37
tonytw1 added 29 commits January 26, 2026 09:02
@tonytw1
Reorder; all vips images ops next to each other.
bb02132
@tonytw1
Put dimensions and orientation operations behind a single interface.
e863843
@tonytw1
Implement dimensions and orientation operations from the same vips im…
bdbebd7 
…age load.
@tonytw1
Put colour model and information operations behind a single interface.
211b5e7
@tonytw1
Put colour model and information operations behind a single interface.
d2e87f6
@tonytw1
Merge colour model and information operations into a single Vips imag…
492dae2 
…e load.
@tonytw1
Put all dimensions, orientation and colour calls under a single method.
da9ed5d
@tonytw1
Merge dimensions, orientation and colour calls under a single vips im…
bb82baf 
…age load.
@tonytw1
Time getImageInformation.
35bb655
@tonytw1
Clean up; dedup thumbnail mime type.
35099d1
@tonytw1
thumbnail javadoc.
7243ef9
@tonytw1
Remove unused imagemagik thumbnail.
62a76cb
@tonytw1
Defer reading of file metadata until after we know we can read the im…
3fa6ffe 
…age.
@tonytw1
Time toFileMetadata.
4e7d170
@tonytw1
Refactor; move thumb dimensions call closed to thumb generation.
d7959ca 
Can thumb generation just tell us this?
@tonytw1
Log to show that we know the dimensions of created thumbnails.
4e89613
@tonytw1
generate thumb nail returns the thumbs dimensions to avoid an immedia…
1903bd5 
…te dimensions call on the upload path.
@tonytw1
Set vips no cache; helps with long running memory?
b490376
@tonytw1
VipsHelper might be less leaky then VipsRaw?
118ffbf
@tonytw1
Move cache = 0 to init.
f95852b
@tonytw1
Log exceptions inside vips.run.
138b7c7
@tonytw1
Manully manage arena to ensure it's explictly closed.
f5fcf2f
@tonytw1
Enable JPEG optimize-coding which appear to be about a 10% file size …
516d558 
…saving for thumbnails.

3751 bytes vs 4067 bytes for a 140x100 thumbnail.

optimize-scans is only useful on progressive jpegs.
@tonytw1
Additional colour interreptations.
707567f
@tonytw1
Add libjemalloc-dev to image loader images.
3edb655
@tonytw1
Select ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so
7d5c18a
@tonytw1
Deprecation: in is deprecated; migrate to slash syntax
f166eb5
@tonytw1
createThumbnail uses the libvips built in srgb profile via the magic …
1d2f88d 
…filename "srgb"
@tonytw1
Correct CMYK renders too light in thumbnails.
0b3e2b3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tonytw1