libobs-d3d12: Add D3D12 graphics backend support

[wanhongqing123]

Description

D3D12 is a new api in windows10 and windows11, Direct3D 12 lets apps get closer to hardware than ever before. By being closer to hardware, Direct3D 12 is faster and more efficient.

• Implement all interfaces in graphics.
• Image、duplicator、video、GDI+ source have been tested.
• WGC Duplication Capture with D3D11 shared texture.

Motivation and Context

As Windows updates and iterates, D3D11 may become obsolete, therefore D3D12 will be supported here.

@Fenrirthviti disagrees with reusing D3D11. The current OBS architecture's rendering pipeline is single-threaded, and multi-threading cannot be implemented at the underlying level. Therefore, D3D12 and D3D11 share some code. I will later place the common methods of D3D11 and D3D12 in appropriate locations.

Furthermore, in the current D3D11 rendering architecture, if the output frame rate is set to 60fps or 120fps, the frame rate will drop when the mouse moves to the middle of the rendered object. With technological advancements, 120fps and 60fps will become necessary. Currently, D3D12 Render and D3D12 NVENC hardware encoding can effectively utilize the advantages of modern GPUs.

Furthermore, NVIDIA graphics cards now support D3D12 encoding mode, so obs-nvenc also needs some modifications.

How Has This Been Tested?

• Video, Image, Color, Duplicator, Game, Media source is teseted.
• Recording is tested.
• Color Range and Space is tested
• NVIDIA's encoder is tested.
• The WinRT Capture section reuses the renderer's device code; however, Microsoft's WinRT Capture may not currently support capture strategies for D3D12 devices. I also consulted some resources but couldn't find a WinRT D3D12 capture implementation. Therefore, this is implemented using a shared texture between D3D11 and D3D12.
• Duplicator Capture not supported by D3D12 Device, this is implemented using a shared texture between D3D11 and D3D12.
  you can use D3D12 by:

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[FerrariHD]

Hi, please dont give up, this feature sounds awesome for modern GPUs! cannot wait to see it in release version

[missingnoXXVI]

Was also too interested for my own good in trying this out and spent a frustrating amount of time getting set up only to get stuck with a NVENC exceeding height/width error that I don't get with my 5060 on the latest OBS Studio's NVENC encoder. Guess I will just have to hope that Wan shows up to continue at a later point.

[wanhongqing123]

Currently, there is a memory leak issue in the main process of NVIDIA's encoding module, and I am working on fixing it.

[missingnoXXVI]

Currently, there is a memory leak issue in the main process of NVIDIA's encoding module, and I am working on fixing it.

👍Glad to see you're still interested then. Was just a but of a mood killer to hit the streaming button only to stuck with the error. For as much as there are additions that I am looking forward to adding to my ease-of-streaming repertoire from the upcoming build of OBS Studio, this has my interest enough that I am fully prepared to stick with it for however long until merged (depending on the stability.)

[wanhongqing123]

This PR can now be used to experience D3D12 features.

[missingnoXXVI]

This PR can now be used to experience D3D12 features.

Haven't tried the latest pr action with the code cleaning, but 2nd to latest I was able to record with the NVENC HEVC D3D12 Encoder quite well for the better part of an hour test. Only thing is now the H.264 D3D12 doesn't show up — unless I am missing that it is currently not included — and it still errors out when I try with the default H.264.

From my log.
12:48:48.397: [obs-nvenc: 'advanced_video_stream'] get_tex_from_handle: OpenSharedResource failed: 0x80070057
12:48:48.397: Error encoding with encoder 'advanced_video_stream'
12:48:48.397: [rtmp stream: 'adv_stream'] Encoder error, disconnecting
12:48:48.397: [rtmp stream: 'adv_stream'] Socket send buffer is 65536 bytes

Still don't have this issue with latest release of OBS Studio or Release Candidate in portable mode. Also can't tell if it is because the newest release I tested fixed it or if recording before helps workaround the problem, but initially when I would try and test stream, after the error out it would also act like it was attempting to record as well but hang and then OBS Studio itself would hang when I would try and exit, forcing me to close it out with Task Manager.

[wanhongqing123]

This PR can now be used to experience D3D12 features.

Haven't tried the latest pr action with the code cleaning, but 2nd to latest I was able to record with the NVENC HEVC D3D12 Encoder quite well for the better part of an hour test. Only thing is now the H.264 D3D12 doesn't show up — unless I am missing that it is currently not included — and it still errors out when I try with the default H.264.

From my log. 12:48:48.397: [obs-nvenc: 'advanced_video_stream'] get_tex_from_handle: OpenSharedResource failed: 0x80070057 12:48:48.397: Error encoding with encoder 'advanced_video_stream' 12:48:48.397: [rtmp stream: 'adv_stream'] Encoder error, disconnecting 12:48:48.397: [rtmp stream: 'adv_stream'] Socket send buffer is 65536 bytes

Still don't have this issue with latest release of OBS Studio or Release Candidate in portable mode. Also can't tell if it is because the newest release I tested fixed it or if recording before helps workaround the problem, but initially when I would try and test stream, after the error out it would also act like it was attempting to record as well but hang and then OBS Studio itself would hang when I would try and exit, forcing me to close it out with Task Manager.

I'm fixing this issue, which is caused by your current rendering device being D3D12, but your encoding device is using D3D11.

[wanhongqing123]

NVIDIA NVENC H264, NVIDIA NVENC HEVC, and NVIDIA NVENC AV1 all currently support D3D12 and D3D11 device encoding. If the rendering device currently being used is a D3D11 device, then D3D11 encoding will be created when encoding is started; otherwise, D3D12 will be used.

[missingnoXXVI]

NVIDIA NVENC H264, NVIDIA NVENC HEVC, and NVIDIA NVENC AV1 all currently support D3D12 and D3D11 device encoding. If the rendering device currently being used is a D3D11 device, then D3D11 encoding will be created when encoding is started; otherwise, D3D12 will be used.

So, if I am reading you correctly, I shouldn't concern myself with distinct D3D12 encoders no longer loading up because renderer usage now should dictate the type of encoding used (D3D11/D3D12) by default?

Either way, I did a quick run through of streaming across Twitch and YouTube (so I could use NVENC AV1, NVENC HEVC, as well as NVENC H.264 again) with the latest attempt and was finally able to stably do so. I intend to do a much lengthier stream and then recording after later but performance seemed great. No hangs or crashes. Switched between Hybrid MP4 and Hybrid MOV to be able to record with the available NVENC selection and that was all 👍👍 as well switching between CBR and CQP. And general performance seems more like making what's already been good for me since I got my 5060 into "even better" is the best way I can describe it.

[wanhongqing123]

NVIDIA NVENC H264, NVIDIA NVENC HEVC, and NVIDIA NVENC AV1 all currently support D3D12 and D3D11 device encoding. If the rendering device currently being used is a D3D11 device, then D3D11 encoding will be created when encoding is started; otherwise, D3D12 will be used.

So, if I am reading you correctly, I shouldn't concern myself with distinct D3D12 encoders no longer loading up because renderer usage now should dictate the type of encoding used (D3D11/D3D12) by default?

Yes, you are right.

[wanhongqing123]

The development of the D3D12 rendering module has been completed. All modules have been tested on my machine and are now ready for code review.

[missingnoXXVI]

The development of the D3D12 rendering module has been completed. All modules have been tested on my machine and are now ready for code review.

No one else has commented on their own testing but I just finished using the last pr action build the same way as I did before and it will be my go to until officially merged or otherwise. Decided to try out the Multi RTMP plugin with it as well and it was smooth sailing.

[Chiitoo]

From a quick look-test-see, still builds and runs on Linux at least. :]

[FerrariHD]

What the chances to see it in Q2 in prod?

[Fenrirthviti]

What the chances to see it in Q2 in prod?

Zero.

[Chiitoo]

What the chances to see it in Q2 in prod?

Seems like something that will take a byte more review than possible to be done for that to happen perhaps.

Nothing stopping from building from source and applying the changes as a patch though!