PNG's with embedded ICC color profile not rendering correctly

[LeonNamowitz]

Describe the bug
For reasons* I have a bunch of 16 bit PNG's with an embedded ICC profile, namely BT.2100 PQ, and those are showing without the color profile applied in Photofield. This means the images just look entirely wrong.
Windows (Photos) and Chrome can render these PNG's properly.

Now from what I gather you never actually render the original file but just the tiles even past full screen but I'm not sure how that works exactly. Maybe it can be fixed with a flag in ffmpeg? Maybe not.

some test files

To Reproduce
Steps to reproduce the behavior:

1. Look at a specified PNG
2. See washed out colors
3. Right click to open the image in a new tab
4. See the proper color managed image (using Chrome)

Expected behavior
Account for embedded color profile in PNG's before rendering.

Screenshots:

Photofield fullscreen

Original image

Metadata

Desktop (please complete the following information):

• OS: Win11 25H2
• Browser: Chrome
• Version: Version 144.0.7559.97 (Official Build) (64-bit)

Additional context
reasons*: The mentioned reasons are that these all originated as HDR screenshots of games using Shadowplay which saves them as .jxr's. But JXR is the most cursed image format on the planet so I'm converting them to PNG + color profile using this.
For example in Immich, this is a problem for the thumbnails as well but once you click on it Immich loads the original file which then shows up correctly.

[SmilyOrg]

Fascinating! That might be why some other formats (RAW, HEIF) look off too. Thanks for the great details!

I'll have a look if it could be a simple ffmpeg fix 🤔

[SmilyOrg]

Ha, I think GitHub is also broken, because the "original" in your post looks dark, not the vivid one I see in my OS image viewer

GitHub	OS	Photofield

[LeonNamowitz]

It's certainly broken for you yea. It looks correct for me though. What browser are you on? Not sure they all support it.

[SmilyOrg]

Ah I was suspecting it could also be a browser thing. Firefox on Fedora, so not completely unexpected 😅

edit: looks fine in Chromium though

[LeonNamowitz]

Well presuming the tiles are rendered using ffmpeg I suppose what's happening is that it just disregards the color profile per default (or assumes BT.709).
I think it should be able to preserve it but I'm not an ffmpeg wizard and no idea how to decide that per PNG. Maybe this trac.ffmpeg.org/wiki/colorspace could help.

[SmilyOrg]

Oof, alright, so I gave it a go and it's quite tricky.

1. Since it's 16 bit, you need to squeeze it into 8 bit somehow (unless you uhh support color profiles through the whole pipeline and that seems ... hard)
2. There are many ways to do that tone mapping and all the other jazz, all of them produce a similar but slightly different results
3. You can do it with ffmpeg and get somewhat close (see below)
4. However, I don't know what doing that does to other images, ideally nothing? What's the impact on decoding speed? Seems like it would need some testing.
5. Most critically, there are many "decoders" (sources) that can be used for each photo in photofield, where ffmpeg is treated more as a last resort (as it's slower than native decoding). So you would have to (selectively?) turn off e.g. the native png decoder (which as you may expect also produces a washed out image), which will make it slow for the cases where calling ffmpeg actually isn't needed. If you don't, the colors will change depending on which encoder happens to be used for a certain zoom level

5 seems a bit of a blocker for this. Maybe a path forward would be to be able to configure sources per-collection? Then you could have a custom source that applies this transform for your case specifically. 4 also seems a bit scary 🙂

Of course, the easiest workaround would be that you reencode your pngs into sRGB / BT.709 with the settings you like and then just view those 😁

Regardless, here's a few things I tried ("original" here means the way Chromium renders it)

1. ffmpeg -hide_banner -loglevel error -i photos/iccpng/Indy_1.jxr.png -vframes 1 -vf "scale='min(iw,1280)':'min(ih,1280)':force_original_aspect_ratio=decrease,zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p" -color_range pc -c:v bmp -pix_fmt rgb24 1.bmp
2. ffmpeg -hide_banner -loglevel error -i photos/iccpng/Indy_1.jxr.png -vframes 1 -vf "scale='min(iw,1280)':'min(ih,1280)':force_original_aspect_ratio=decrease,zscale=t=linear:npl=180,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=hable:desat=0,zscale=t=bt709:m=bt709,format=yuv420p" -color_range pc -c:v bmp -pix_fmt rgb24 2.bmp
3. ffmpeg -hide_banner -loglevel error -i photos/iccpng/Indy_1.jxr.png -vframes 1 -vf "scale='min(iw,1280)':'min(ih,1280)':force_original_aspect_ratio=decrease,zscale=t=linear:npl=180,format=gbrpf32le,zscale=p=bt709,tonemap=tonemap=reinhard:desat=0,zscale=t=bt709:m=bt709,format=yuv420p" -color_range pc -c:v bmp -pix_fmt rgb24 3.bmp
4. ffmpeg -hide_banner -loglevel error -i photos/iccpng/Indy_1.jxr.png -vframes 1 -vf "scale='min(iw,1280)':'min(ih,1280)':force_original_aspect_ratio=decrease,zscale=t=linear:npl=100,format=gbrpf32le,zscale=p=bt709,tonemap=mobius:param=0.3:desat=0,zscale=t=bt709:m=bt709:r=tv,format=yuv420p" -pix_fmt rgb24 -c:v bmp 4.bmp
5. ffmpeg -hide_banner -loglevel error -i photos/iccpng/Indy_1.jxr.png -vframes 1 -vf "scale='min(iw,1280)':'min(ih,1280)':force_original_aspect_ratio=decrease,zscale=t=bt709:p=bt709:m=bt709" -c:v bmp -pix_fmt rgb24 5.bmp
6. ffmpeg -hide_banner -loglevel error -i photos/iccpng/Indy_1.jxr.png -vframes 1 -vf "scale='min(iw,1280)':'min(ih,1280)':force_original_aspect_ratio=decrease,zscale=t=linear:p=bt709,tonemap=hable:param=1.0:desat=0,zscale=t=iec61966-2-1:p=bt709:r=full" -c:v bmp -pix_fmt rgb24 6.bmp

[LeonNamowitz]

I don't exactly get your point about having to quantize to 8 bit. Forgive me if I'm wrong somewhere but PNG supports 48bpp / 16 bit per channel natively, without ICC profile. The profile mainly changes the primaries to match BT.2020 or the like right?

These 2 random images I've found are 16 bit PNG's and show entirely correctly on my Photofield as far as I can tell. While the other 2 from link 1 with ICC profiles break again.
NVM they have expected color banding as even the fullscreen tiles of those are just 8 bit JPG's.

link 1 - sRGB file	link 2

---

Right so I guess in fullscreen or zoomed past 100% it's the native decoder then? It's kind of hard to tell with the debug overlay at that zoom. For that one I don't even know where to start.

And yep point 4 and 5 is what I was concerned about as well. How to decide which PNG's to treat differently at what zoom and so on.

Maybe a path forward would be to be able to configure sources per-collection?

This doesn't help me much because it's not always only those in a folder..

Of course, the easiest workaround would be that you reencode your pngs into sRGB / BT.709 with the settings you like and then just view those 😁

Maybe I should just do that but it sounds like work for my whole collection and technically I'm getting better colors like this ^^

Honestly this whole thing was kind of a long shot problem anyway and it doesn't bother me much. So I'm alright if nothing else comes of this. Thanks though.

[SmilyOrg]

I don't exactly get your point about having to quantize to 8 bit. Forgive me if I'm wrong somewhere but PNG supports 48bpp / 16 bit per channel natively, without ICC profile. The profile mainly changes the primaries to match BT.2020 or the like right?

To be honest, I don't know too much about how color spaces work. I believe the much of the stdlib image code in Go actually supports up to 32 bit, but then this likely gets quantized down to 8 bit once I composite & encode it into jpeg/webp tiles.

My best understanding is that the screenshots are actually HDR, and the color space is set up to use those 16 bits most effectively for that purpose. That means that if you just straight up quantize 16 to 8 without any tone mapping, you'll get a washed out looking image.

These 2 random images I've found are 16 bit PNG's and show entirely correctly on my Photofield as far as I can tell. While the other 2 from link 1 with ICC profiles break again.
link 1 - sRGB file link 2

I'm guessing in these cases you might only see a difference if you had a 10/12 bit monitor (somewhat uncommon?) and even then it might be tricky.

Right so I guess in fullscreen or zoomed past 100% it's the native decoder then? It's kind of hard to tell with the debug overlay at that zoom. For that one I don't even know where to start.

Most likely yes. It's also a little hard for me to say what will happen in a specific scenario, as the decoder selection is somewhat dynamic, mostly controlled by how long it is predicted for the decoding to take and what decoders are even available for a certain image (e.g. djpeg won't work for pngs). This is controlled by the various timings configured in the "Media source configuration".

For PNGs specifically, as you zoom in, you'll likely start by using the internal thumbnail (sqlite), and then transition to native (image), unless native fails, then it would be ffmpeg.

But it's very possible that the logic is broken in some cases, it's a little hard to tune :)

And yep point 4 and 5 is what I was concerned about as well. How to decide which PNG's to treat differently at what zoom and so on.

Maybe a path forward would be to be able to configure sources per-collection?

This doesn't help me much because it's not always only those in a folder..

Yeah I was guessing that might also be the case.

Of course, the easiest workaround would be that you reencode your pngs into sRGB / BT.709 with the settings you like and then just view those 😁

Maybe I should just do that but it sounds like work for my whole collection and technically I'm getting better colors like this ^^

Hmm, I'm interested in whether you have a monitor that can take advantage of it and e.g. Chrome knows how to display it properly? Because on my (quite old) very normal 1440p 8-bit non-HDR Dell, it will look very much the same either way 😅

I do have a Pixel phone that supports HDR though, so I was itching to add support for Google's HDR implementation that uses an extra luminance layer embedded into JPEG, but that sounds like a lot of work to support tbh 😁

Honestly this whole thing was kind of a long shot problem anyway and it doesn't bother me much. So I'm alright if nothing else comes of this. Thanks though.

It might need a little more cooking time yeah, but thanks for the interesting discussion so far anyway!

Thanks for the interesting

[LeonNamowitz]

To the first point:
I thought about it some more and you're totally right. You have to tonemap somewhere (which is already a mess).
I also just realized the tiles are just jpeg's so I looked at my 16 bit test PNG's again and there is in fact quite a lot of banding.
I am just blind. Please disregard.

For PNGs specifically, as you zoom in, you'll likely start by using the internal thumbnail (sqlite), and then transition to native (image), unless native fails, then it would be ffmpeg.

For me it pretty much goes from sqlite -> ffmpeg -> ffmpeg -> "original" (which I imagine is the native decoder?) while zooming in.
But I have tweaked the config quite a bit and the files are on a HDD currently so it's probably expected.

I'm guessing in these cases you might only see a difference if you had a 10/12 bit monitor (somewhat uncommon?) and even then it might be tricky.

I don't know how uncommon that is these days. Definitely not on the high end since that moved to OLED and all of those are 10-bit. I have a MSI QD-OLED and I can certainly tell but I am also very particular about image quality ;)

[SmilyOrg]

You got me on the monitors, I am severely out of date there. I should probably update to something that's less than a decade old soon™️, but there is just... so much choice 🤷‍♂️