New SSAO Algorithm by Monsterovich · Pull Request #62 · xycaleth/OpenJK

Monsterovich · 2018-05-27T18:10:19Z

This is continuation of my previous failed experiments with AO. I rewrote ssao.glsl completely, it looks good IMO.

Issues:

AO affects player models a bit. (TBH this is present in a lot of games, idk what to do with it) I like it very much anyway.

Video:
https://www.youtube.com/watch?v=1tx_PHvTj0g

Screenshots:
r_ssao 0

r_ssao 1

Droideka best class (C)

xycaleth · 2018-05-27T21:13:41Z

What improvements does this have over the existing SSAO implementation?

I have a few issues with your implementation:

I'm really not liking the black outline that this implementation gives around everything. This is really noticeable around the sides of the ship in the lighter textured map.
it just looks like the image is made darker when ever there is a depth discontinuity, regardless of whether the object is already lit or not.

Monsterovich · 2018-05-27T22:02:48Z

What improvements does this have over the existing SSAO implementation?

It looks smoother and doesn't have flickering (!). The existing SSAO looks even worse while moving.
When I tried it for the first time, I thought it was a bug.

Old SSAO:

New SSAO:

The new SSAO doesn't have this garbage on corners and walls.
Old:

New:

Monsterovich · 2018-05-27T22:10:08Z

I'm really not liking the black outline that this implementation gives around everything. This is really noticeable around the sides of the ship in the lighter textured map.

Perhaps, as I said earlier such effect is present in many games, I would even like to keep it (because players look more intensive) but I should improve it first. (and make ssao radius configurable)

it just looks like the image is made darker when ever there is a depth discontinuity, regardless of whether the object is already lt or not.

I didn't notice anything. Can you please screenshot something?

Monsterovich · 2018-05-28T10:40:42Z

Improved my shader a lot:

Frame 1: NoSSAO (r_ssao 0):
Frame 2: SSAO (r_ssao 1, r_ssao_lightmap 0): This doesn't make the image darker as you wanted. :P
Frame 3: SSAO + applying lightmap (r_ssao 1, r_ssao_lightmap 2.2360679): The existing formula from my shader just improved it.

xycaleth · 2018-05-28T11:13:29Z

Firstly, I will say that the map you've chosen in your screenshots probably isn't the best one to use to show off SSAO. I don't think there's anything in the screenshot which would benefit from having SSAO... you probably want something which has a few players and less light sources.

Perhaps, as I said earlier such effect is present in many games, I would even like to keep it (because players look more intensive) but I should improve it first. (and make ssao radius configurable)

No, please don't keep it. Game developers are actively trying to avoid exactly this dark outline which older/naïve SSAO algorithms would give. It looks terrible and is in no way an accurate representation of ambient occlusion.

I didn't notice anything. Can you please screenshot something?

Here's part of your without SSAO image: it's the rectangle just to the right of the "Spectator" text.

I'm mainly pointing out the floor - the gradient there comes from the lightmap. Should adding ambient occlusion affect that? I don't think so, the area is fairly well lit and nothing is occluding it.

Here's with your SSAO:

There's two problems I have with this:

The whole image is now darker. Why? What is occluding the top of the ship? Not in the screenshot, but this is especially noticeable in front of the cockpit on top of the ship.
There's a thick black outline along the edge of the ship, when it doesn't make sense to have one (it never makes sense to have a black outline on anything unless you're rendering a cartoon...).

This is still a problem in your new screenshots too. This is the top left corner of the image:

Without SSAO:

With SSAO:

There shouldn't be a black outline where there's a depth discontinuity.

I appreciate your help, but I don't think this is the right direction to take - if the problem with the existing SSAO algorithm is that it flickers when you move, and has darkening on corners, then I think those should be fixed instead of replacing it entirely with something else.

Monsterovich · 2018-05-28T18:06:56Z

Done.

Monsterovich · 2018-05-29T11:28:59Z

There is one problem left and I can't fix it.

Seems SSAO is rendered after the fog. How does the fog work in rend2?

SomaZ · 2018-05-30T11:21:24Z

Ssao is rendered before the fog actually, but the ssao buffer simply gets stamped on the render buffer after everything has been rendered. This will also result in bad ssao when transparent surfaces were rendered. Fog will be applied as a separate stage per object right now. Solution for this problem could be applying the ssao before rendering fog stages and transparent objects. The other solution which I would personally prefer, is passing the ssao buffer to the lightall shader and apply it only to the ambient term of the light and remove the blit of the ssao buffer.

Monsterovich · 2018-05-30T22:27:34Z

I don't know this part of code well, you can merge these commits and fix it yourself. :P This is the best solution.

Monsterovich · 2018-06-01T11:29:16Z

Or you can give me some tips how to do this and I'll try.

xycaleth · 2018-06-01T12:12:21Z

Sorry haven’t had time to review in full. I think it’s starting to look good but I have a few suggestions regarding your sampling pattern. Will try to reply this weekend

xycaleth

So I pulled your code and had a play about with the SSAO. For now I wouldn't worry about the fog problem - I imagine it's a problem with the existing implementation as well so it's not like it's a step backwards. That can be fixed after the code gets merged.

Anyhow, at a quick glance the SSAO looks good, but looking closer I can see some room for improvement. Most of my comments are about the code, but there's two points about the algorithm itself I want to mention:

You compare the linear depth values in readDepth but the difference between znear and zfar is dynamic based on what you can see. i.e. if zfar is really big, then the SSAO will be lighter because the linear depth values are spread over a larger distance. When zfar is really close, SSAO will be darker. I think instead of comparing depth values, the real distances need to be compared. This would involve converting the depth value back into world or view-space positions
The pattern used for sampling depths is just sampling left/right/up/down, 4 samples in each direction at 1, 2, 4, and 8 intervals. This produces a weird pixelated effect if you look closely and you move the camera slightly. I think a pattern which samples in multiple directions around a circle will look better. I think this is what the Poission disc array was for in the existing implementation but perhaps it wasn't done very well. Anyway, I'm not saying the number of samples need to be increased, but the position of the samples needs to be more circular.

xycaleth · 2018-06-02T09:56:01Z

codemp/rd-rend2/glsl/ssao.glsl

Can znear and zfar values be passed in as function parameters? I'd like to avoid using global variables unless it's necessary.

xycaleth · 2018-06-02T09:56:10Z

codemp/rd-rend2/glsl/ssao.glsl

This seems unnecessary

xycaleth · 2018-06-02T09:57:35Z

codemp/rd-rend2/glsl/ssao.glsl

All the GLSL shaders in rend2 are compiled with version "GLSL 1.50 Core". This means the texture2D function is unavailable - I tried on my laptop and this shader fails to compile.

texture2D should be replaced with texture instead.

xycaleth · 2018-06-02T09:58:34Z

codemp/rd-rend2/glsl/ssao.glsl

Pass in znear and zfar as arguments

xycaleth · 2018-06-02T10:05:25Z

codemp/rd-rend2/glsl/ssao.glsl

+
 	int i;
-	for (i = 0; i < 3; i++)
+	for (i = 0; i < 4; i++)


Only sampling depths sideways and up/down doesn't look very good - it almost looks kind of pixelated. I think this needs to be changed to sampling in directions all around the currently pixel, not just left/up/down/right.

This is an awful lot of samples now... you're doing 64 samples now. Can you reduce this down to 16 again (like you had before)?

xycaleth · 2018-06-02T10:25:11Z

codemp/rd-rend2/glsl/ssao.glsl

Remove the comment - it's already obvious from the function name that it generates random numbers.

What's not obvious is the range of numbers that the function can output. Does the function output random numbers between 0 and 1? 0 and 0.004? Perhaps a comment can be added to explain that instead

xycaleth · 2018-06-02T10:28:13Z

codemp/rd-rend2/glsl/ssao.glsl

Oops. Changed it to 1.0.

xycaleth · 2018-06-02T10:30:47Z

codemp/rd-rend2/tr_init.cpp

+	r_ssao_aocap = ri.Cvar_Get( "r_ssao_aocap", "1.5", CVAR_ARCHIVE, "" );
+	r_ssao_strength = ri.Cvar_Get( "r_ssao_strength", "1.0", CVAR_ARCHIVE, "" );
+	r_ssao_aoMultiplier = ri.Cvar_Get( "r_ssao_aoMultiplier", "20000.0", CVAR_ARCHIVE, "" );
+	char val[32]; sprintf(val, "%f", std::sqrt(10));


Why sqrt(10)???

xycaleth · 2018-06-02T10:32:51Z

codemp/rd-rend2/tr_init.cpp

How are strength and aoMultiplier different?

Forgot to remove. I added it for testing purposes.

xycaleth · 2018-06-02T10:33:18Z

codemp/rd-rend2/tr_init.cpp

+	r_ssao_strength = ri.Cvar_Get( "r_ssao_strength", "1.0", CVAR_ARCHIVE, "" );
+	r_ssao_aoMultiplier = ri.Cvar_Get( "r_ssao_aoMultiplier", "20000.0", CVAR_ARCHIVE, "" );
+	char val[32]; sprintf(val, "%f", std::sqrt(10));
+	r_ssao_lightmap = ri.Cvar_Get( "r_ssao_lightmap", val, CVAR_ARCHIVE, "" );


What's this for?

Monsterovich · 2018-06-02T14:26:09Z

Only sampling depths sideways and up/down doesn't look very good - it almost looks kind of pixelated. I think this needs to be changed to sampling in directions all around the currently pixel, not just left/up/down/right.

Done.

SSAO Pixel (example):

Monsterovich · 2018-06-02T16:51:46Z

Question: does MSAA apply on SSAO in rend2?

xycaleth · 2018-06-02T16:58:27Z

No, it doesn’t make sense to apply MSAA to SSAO or any of the post processing effects. It only works on aliasing from geometry edges

Monsterovich · 2018-06-02T22:21:17Z

I guess, everything is done except bug with fog and translucent areas.

xycaleth · 2018-06-02T22:33:06Z

There's still a few comments you haven't addressed... but going to go through everything again now anyway

xycaleth · 2018-06-02T22:37:06Z

codemp/rd-rend2/glsl/ssao.glsl

+
+	float d;
+
+	float pw = 1.0 / u_ScreenInfo.x;


I don't think you need pw and ph. If you use texelFetch to read from the texture, you can use texel coordinates:

ivec2 screenCoord = ivec2(g_FragCoord.xy); vec4 d; d = texelFetch(u_ScreenDepthMap, screenCoord, 0).x; d = texelFtech(u_ScreenDepthMap, ivec2(screenCord.x, screenCoord.y + 1, 0).x; etc..

If you really want to continue using texture, then u_ScreenInfo should contain (1.0 / w, 1.0 / h) instead of doing it in the shader.

xycaleth · 2018-06-02T22:38:51Z

codemp/rd-rend2/glsl/ssao.glsl

+
 	int i;
-	for (i = 0; i < 3; i++)
+	for (i = 0; i < 4; i++)


This is an awful lot of samples now... you're doing 64 samples now. Can you reduce this down to 16 again (like you had before)?

xycaleth · 2018-06-02T22:39:36Z

codemp/rd-rend2/glsl/ssao.glsl

Divisions are generally slower than multiplications.

This can be changed to..

ao += compareDepths(...) * aoScale; ... aoScale *= 0.8;

xycaleth · 2018-06-02T22:40:23Z

codemp/rd-rend2/glsl/ssao.glsl

+		d = readDepth( vec2(var_ScreenTex.x,var_ScreenTex.y+ph), u_ViewInfo.x, u_ViewInfo.y);
+		ao += compareDepths(depth,d,u_ViewInfo.x,u_ViewInfo.y) / aoScale;
+
+		pw *= 2.0;


Why are you multiplying pw and ph by 2?

xycaleth · 2018-06-02T22:41:10Z

codemp/rd-rend2/glsl/ssao.glsl

-	float result = ambientOcclusion(u_ScreenDepthMap, var_ScreenTex, u_ViewInfo.x, u_ViewInfo.y);
-
-	out_Color = vec4(vec3(result), 1.0);
+	ao /= 32.0;


I'm guessing this should be divided by the number of samples you've taken (which is 64 at the moment)

xycaleth · 2018-06-02T22:41:33Z

codemp/rd-rend2/tr_glsl.cpp


 	qglUseProgram(tr.ssaoShader.program);
 	GLSL_SetUniformInt(&tr.ssaoShader, UNIFORM_SCREENDEPTHMAP, TB_COLORMAP);
+	GLSL_SetUniformInt(&tr.ssaoShader, UNIFORM_SCREENIMAGEMAP, TB_LIGHTMAP);


Don't need this anymore

xycaleth · 2018-06-02T22:42:14Z

codemp/rd-rend2/tr_init.cpp


+	r_ssao_aocap = ri.Cvar_Get( "r_ssao_aocap", "1.0", CVAR_ARCHIVE, "" );
+	r_ssao_aoMultiplier = ri.Cvar_Get( "r_ssao_aoMultiplier", "20000.0", CVAR_ARCHIVE, "" );
+	char val[32]; sprintf(val, "%f", std::sqrt(10));


Why have you used sqrt(10)? What's so special about that number?

xycaleth · 2018-06-02T22:42:52Z

codemp/rd-rend2/tr_init.cpp

+	r_ssao_aocap = ri.Cvar_Get( "r_ssao_aocap", "1.0", CVAR_ARCHIVE, "" );
+	r_ssao_aoMultiplier = ri.Cvar_Get( "r_ssao_aoMultiplier", "20000.0", CVAR_ARCHIVE, "" );
+	char val[32]; sprintf(val, "%f", std::sqrt(10));
+	r_ssao_lightmap = ri.Cvar_Get( "r_ssao_lightmap", val, CVAR_ARCHIVE, "" );


What does this cvar do? It looks like you're multiplying the result of the shader by this value.. but you already have a multiplier cvar (aoMultiplier)

Monsterovich · 2018-06-02T22:59:47Z

...and SSAO flickering is not resolved but I noticed that the same thing happens with glow.

https://youtu.be/jo7oPZ4k0Tg

Monsterovich · 2018-06-03T17:04:21Z

I think we should find the reason of flickering first. I can't configure SSAO with that problem present in rend2.

Monsterovich · 2018-06-08T17:09:05Z

https://www.youtube.com/watch?v=8d41OgfXZ4Y&feature=youtu.be

Can you please help me with flickering of glow and ssao?

SomaZ · 2018-06-08T17:25:56Z

Well, its two different problems from the same origin. Both algorithims use downscaled buffers. For the ssao you could try increasing the buffer size to match the viewport size and check if the flickering disapears. Look in tr_image.cpp for the "tr.screenSsaoImage = ... " and remove the division for width and height. This should be enough. You could try the same with tr.quarterImage[x] which is used as buffer for the ssao scaling. For the glow you can simply reduce the passes via cvar to 4 or something and look if its still appealing to you.

Monsterovich · 2018-06-08T20:50:32Z

Well, its two different problems from the same origin. Both algorithims use downscaled buffers. For the ssao you could try increasing the buffer size to match the viewport size and check if the flickering disapears.

I've noticed that when I worked on SSAO shader. I changed buffer size for a test but the problem was still there.

Look in tr_image.cpp for the "tr.screenSsaoImage = ... " and remove the division for width and height. This should be enough. You could try the same with tr.quarterImage[x] which is used as buffer for the ssao scaling. For the glow you can simply reduce the passes via cvar to 4 or something and look if its still appealing to you.

I don't know how this should help but when I did that the whole screen started to flicker. :D

Monsterovich · 2018-06-09T10:23:34Z

Well, https://www.youtube.com/watch?v=fH8xLlcZ5uE look at this video and you will see that SSAO image is flickering (horizontal and vertical lines) even if it doesn't match the viewport.

The problem is not with the viewport anyway I think.

Monsterovich · 2018-06-18T00:03:28Z

The flickering is also present in vanilla renderer. I'm curious now what it could be?

Monsterovich added 4 commits May 27, 2018 19:55

Added cool AO shader

2744651

Merge remote-tracking branch 'xycaleth/rend2' into rend2-newao

38dbbca

Removed unnecessary complications

6ad9c1f

Merge remote-tracking branch 'xycaleth/rend2' into rend2-newao

5e725ac

Merge branch 'rend2' into rend2-newao

26b6c4e

Monsterovich added 2 commits May 28, 2018 02:22

Added CVARs to configure shader

f81afaa

Fix

64fcc23

Improve SSAO algorithm

fc0091e

Fixed AO halos and flat surface darkening

b86d4b4

Merge remote-tracking branch 'xycaleth/rend2' into rend2-newao

3cba753

xycaleth requested changes Jun 2, 2018

View reviewed changes

Monsterovich added 3 commits June 2, 2018 15:05

Clean code

41c7243

Replace Texture2D to Texture

d12fa4c

Improved SSAO rendering

4c44001

Monsterovich added 2 commits June 2, 2018 21:27

Improved pixel circle

addfdbd

Removed unused variable

e78dec8

Removed unused code

f08a371

xycaleth requested changes Jun 2, 2018

View reviewed changes

Fixed circle formula

b3936d0

Replace / with *

d888306

Merge remote-tracking branch 'xycaleth/rend2' into rend2-newao

3d04b09

Conversation

Monsterovich commented May 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xycaleth commented May 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Monsterovich commented May 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Monsterovich commented May 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Monsterovich commented May 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xycaleth commented May 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Monsterovich commented May 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Monsterovich commented May 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SomaZ commented May 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Monsterovich commented May 30, 2018

Uh oh!

Monsterovich commented Jun 1, 2018

Uh oh!

xycaleth commented Jun 1, 2018

Uh oh!

xycaleth left a comment

Choose a reason for hiding this comment

Uh oh!

xycaleth Jun 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Monsterovich commented Jun 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Monsterovich commented Jun 2, 2018

Uh oh!

Monsterovich commented May 27, 2018 •

edited

Loading

xycaleth commented May 27, 2018 •

edited

Loading

Monsterovich commented May 27, 2018 •

edited

Loading

Monsterovich commented May 27, 2018 •

edited

Loading

Monsterovich commented May 28, 2018 •

edited

Loading

xycaleth commented May 28, 2018 •

edited

Loading

Monsterovich commented May 28, 2018 •

edited

Loading

Monsterovich commented May 29, 2018 •

edited

Loading

SomaZ commented May 30, 2018 •

edited

Loading

xycaleth Jun 2, 2018 •

edited

Loading

Monsterovich commented Jun 2, 2018 •

edited

Loading

Monsterovich commented Jun 2, 2018 •

edited

Loading

Monsterovich commented Jun 8, 2018 •

edited

Loading

Monsterovich commented Jun 9, 2018 •

edited

Loading