Filter chain loading locks up with parallel compilation in D3D11 when running under ReShade

[josejl1987]

Hello, and thank you for making this!

I've been developing a ReShade addon that enables the use of librashader with the output from Lossless Scaling. However, I've encountered some issues that I hope to resolve.

1. Shader Behavior Differences Between Runtimes

Some presets from the Sonkun CRT Guest Advanced HD shader pack are behaving very differently across different graphics APIs:

D3D12 Output:

Vulkan & D3D11 Output:

It can be reproduced with the CLI executable as well.

2. D3D11 Parallel Shader Compilation Issue

Additionally, I had to disable parallel shader compilation for D3D11. When running the addon from a DLL, concurrent calls to ID3D11Device::CreateInputLayout get stuck. Interestingly, this issue does not occur when running the same code from a standalone EXE.

[chyyran]

I was able to reproduce this somewhat on D3D11 with shaders pared down to just pre-shaders-afterglow-grade.slang. I wasn't able to reproduce on Vulkan however.

Incorrect output with D3D11:

Correct output with D3D12:

However when I looked at the RenderDoc dumps for these two, I wasn't able to find any conclusive information. I'll need some more followup items to dig further into it.

1. What specific shader preset are you using to test?
2. RenderDoc dumps for D3D12, Vulkan, and D3D11.
3. Does this shader preset work in RetroArch on the Vulkan, D3D11, and OpenGL3 runtime?

As for the parellel shader compilation issue, I'll bring that up with some DirectX folks to see if they have any ideas around it. The ideal way to debug this would be to attach Visual Studio to it and see if you get any debug output or stack traces from D3D11 though, if you're able to enable D3D11_CREATE_DEVICE_FLAG somehow.

[chyyran]

From Kaldaien on the DirectX discord

ReShade unfortunately has a lot of SRW locks where the actual D3D11 API uses Critical Sections 😕
I've seen hook recursion caused by interactions between it and other third-party render mod tools cause deadlock.
You might as well build ReShade from source, it'll help you figure out what the heck's going on 🙂
Those problems usually only manifest in heavily hooked parts of the API though, related to rendering overlays.
CreateInputLayout seems... trivial.
But it could be ReShade's AddOn state tracking system to blame, since it covers the entire API and forwards them to any addon that will listen for them.

[josejl1987]

Oops, never mind about Vulkan. I tried the Vulkan triangle sample, and it works fine there. For some reason, the CLI util is not outputting anything if I select Vulkan, no error or anything, but that looks like a separate issue.

For example, it happens with sonkun\aperture-grille\flat-screen\1080p\02-1080p-crt-guest-advanced-hd-aperture-grille-u-normal-composite, I get the same output as the screenshot you posted.

Does it still make sense to send Renderdoc captures? I can't really capture the addon, since ReShade isn't captured when you run Renderdoc.
I tried to compare your D3D11 and D3D12 triangle samples, but I think there is no way to get the debug symbols when doing the SPIRV to DXIL conversion? I am not too experienced with either D3D12 or Rust though, so I might have missed something.

As for the parallel compilation, I managed to narrow down it a little, it seems to get locked with shaders that load LUTs. But I'll try to get more information with the Reshade debug DLL.

[chyyran]

According to https://reshade.me/forum/addons-discussion/8640-renderdoc-and-reshade the ReShade test application should be able to provide a RenderDoc capture.

SPIR-V to DXIL does not have debug symbols unfortunately. The conversion there is quite manual and doesn't support outputting debug symbols.

[chyyran]

Confirmed this is a shader issue, reported upstream

(this might be another excuse to take another look at spirv-to-dxbc..)

[chyyran]

As for the parallel compilation, I managed to narrow down it a little, it seems to get locked with shaders that load LUTs. But I'll try to get more information with the Reshade debug DLL.

Have you tried using filter_chain_create_deferred to defer loading LUTs?

[josejl1987]

Yes, I was using that right away. If I use the regular version, it gets stuck in ID3D11Device::CreateVertexShader instead of CreateInputLayout.

As far as I can tell, this is using the real D3D11 code, bypassing ReShade hooks.

[chyyran]

guest.r was able to fix the faulty pre-shaders-afterglow-grade shader on D3D11.

I’ll keep looking into the parallel shader issue but it looks like it is interacting poorly with ReShade’s locks.

[chyyran]

Are you able to provide a debug dump file at the point where it hangs?

[josejl1987]

main.dmp

[chyyran]

Would it be possible to provide a heap dump (minidump with heap) and PDBs for reshade_librashader.addon and the build (or verison) of librashader.dll you're using? Without the heap its a bit hard to see what if anything is holding the lock.