Add AV1 NVENC Ultra High Quality Mode (RTX 5080)/ svt-av1 Hybrid

[jonrodrigos]

Hi Ragaya, how are you?
I was reading the latest NVIDIA documentation about NVENC AV1 on the new RTX 50 series GPUs, and I saw that they added a mode called Ultra High Quality.
According to NVIDIA, this mode is designed to preserve film grain with cinematic quality, ensuring superior visual fidelity for HDR and Dolby Vision content.
In HEVC we already have the --tune uhq command, but in AV1 this parameter doesn’t work. That’s why I’d like to ask: would it be possible for you to add support in StaxRip/NVEnc for this new Ultra High Quality mode in AV1 for the RTX 5080? Maybe it could be exposed as a new flag, something like:

• --preset ultra-high-quality
• or --tune uhq-av1
  In addition, I’d like to know if there is any plan to implement support for unidirectional B‑frames in AV1 NVENC. From what I’ve read, they provide benefits not only in compression efficiency but also in preserving fine details and film grain.
  Another point: I noticed that you added support for SVT‑AV1 within StaxRip/NVEnc, but I couldn’t find clear documentation explaining how to enable or use this feature properly. Could you clarify how the integration works and which parameters we should use?
  I believe these additions would be very valuable for anyone seeking maximum preservation of texture and detail in movies and series, especially in HDR/Dolby Vision content.
  Thanks a lot for your attention!

[rigaya]

AV1 UHQ mode

Being duplicate of #709, it can be assumed that it is near to adding -b 7 --tf-level 4 --lookahead 32 --lookahead-level 3 --no-i-adapt --no-b-adapt --bref-mode middle.

https://developer.nvidia.com/blog/nvidia-video-codec-sdk-13-0-powered-by-nvidia-blackwell/

In the UHQ tuning, the optimal settings for lookahead and temporal filtering are applied in combination rather than individually. As with UHQ HEVC, the number of B-frames is set to five, while using the middle B-frame as the reference. UHQ mode also disables adaptive I- and B-frames and uses a fixed GOP (group of pictures) structure.

For AV1, the UHQ tuning info in NVIDIA Blackwell has seven B-frames, instead of five, further enhancing the compression efficiency.

SVT-AV1 support

You can see examples here, although I'm not bright in detailed option of each specific encoder, you shall be able to pass params by --avcodec-prms.
https://github.com/rigaya/NVEnc/blob/master/NVEncC_Options.en.md#--avcodec-prms-string

[rigaya]

NVEnc 9.05 adds --tune uhq for convenience.

[jonrodrigos]

Hello, Rigaya,
Thank you for implementing the HQ tuning feature — I’ll be running some tests soon. Apologies for the delay in getting back to you.
Regarding SVT-AV1, I haven’t been able to get it working. I tried running it directly via CMD, through the custom section under NVEnc in StaxRip, and also via the command line field. When I include -c av_libsvtav1, it throws various errors. If I remove that part, the process continues, but it seems to only use the GPU — SVT-AV1 doesn’t appear to be active.
Here’s the command I used, based on your example:
-c av_libsvtav1 --avcodec-prms "preset=6,crf=30,svtav1-params=enable-variance-boost=1:variance-boost-strength=2"
If you have any tips on how to make this work properly in StaxRip, I’d really appreciate it.
From what I understand, this feature is a hybrid between GPU and CPU usage — if that’s the case, it would be extremely helpful for movie conversions.

[rigaya]

Would you please share the log of the errors to know where the problem is?

Example below is the most simple way to encode using libsvtav1 from NVEnc.

x64\NVEncC64.exe -i input.mp4 -o F:\temp\test.mp4 -c av_libsvtav1 --avcodec-prms "preset=6,crf=30,svtav1-params=enable-variance-boost=1:variance-boost-strength=2"

If you want raw output, you might need to add -f ivf.

x64\NVEncC64.exe -i input.mp4 -o F:\temp\test.av1 -f ivf -c av_libsvtav1 --avcodec-prms "preset=6,crf=30,svtav1-params=enable-variance-boost=1:variance-boost-strength=2"

Below is example log for the first command line used.

--------------------------------------------------------------------------------
F:\temp\test.mp4
--------------------------------------------------------------------------------
Svt[info]: -------------------------------------------
Svt[info]: SVT [version]:       SVT-AV1 Encoder Lib v3.1.0
Svt[info]: SVT [build]  :       GCC 15.2.0       64 bit
Svt[info]: LIB Build date: Aug 24 2025 09:27:59
Svt[info]: -------------------------------------------
Svt[info]: Level of Parallelism: 6
Svt[info]: Number of PPCS 305
Svt[info]: [asm level on system : up to avx512icl]
Svt[info]: [asm level selected : up to avx512icl]
Svt[info]: -------------------------------------------
Svt[info]: SVT [config]: main profile   tier (auto)     level (auto)
Svt[info]: SVT [config]: width / height / fps numerator / fps denominator               : 1920 / 1080 / 30000 / 1001
Svt[info]: SVT [config]: bit-depth / color format                                       : 8 / YUV420
Svt[info]: SVT [config]: preset / tune / pred struct                                    : 6 / PSNR / random access
Svt[info]: SVT [config]: gop size / mini-gop size / key-frame type                      : 161 / 32 / key frame
Svt[info]: SVT [config]: BRC mode / rate factor                                         : CRF / 30
Svt[info]: SVT [config]: AQ mode / Variance Boost strength / octile / curve             : 2 / 2 / 6 / 0
Svt[info]: SVT [config]: sharpness / luminance-based QP bias                            : 0 / 0
Svt[info]: -------------------------------------------
NVEncC (x64) 9.05 (r3460) by rigaya, Oct 22 2025 10:37:43 (VC 1944/Win)
OS Version     Windows 11 x64 (26200) [UTF-8]
CPU            AMD Ryzen 9 7950X 16-Core Processor [5.51GHz] (16C/32T)
GPU            #1: NVIDIA GeForce RTX 4080 (9728 cores, 2505 MHz)[PCIe4x16][581.42]
CUDA           CUDA 13.0, schedule mode: auto
Input Buffers  CUDA, 16 frames
Input Info     avcuvid: H.264/AVC, 1920x1080, 30000/1001 fps
Vpp Filters    cspconv(nv12 -> yv12)
               copyDtoH


encoded 4550 frames, 96.50 fps, 2849.66 kbps, 51.57 MB
encode time 0:00:47, CPU: 54.9%, GPU: 41.3%, VD: 14.6%, GPUClock: 670MHz, VEClock: 1343MHz
frame type IDR 4550
frame type I   4550,  total size  51.57 MB

[johnvick]

Thanks for adding --tune uhq - specific to AV1, what settings does this replace?

[rigaya]

I’m not explicitly changing the values in the application, therefore not sure for each specific values. Instead, the driver is replacing various options for you when using --tune uhq to optimistic value. This should be what is expected which is described in NVIDIA documentation.

[jonrodrigos]

Hi Ragaya,
Sorry for bothering you so much with this, but I really haven’t been able to get it working.
The idea of using SVT-AV1 with NVEnc is fantastic because it enables a hybrid approach — using the CPU for AV1 encoding while keeping the GPU free for other tasks during conversion, like upscaling, SDR to HDR conversion, etc. That would be a huge help in optimizing the workflow.
I followed the examples you provided, tested via CMD, through the custom section in NVEnc within StaxRip, and also through the command line field — but I keep getting the same errors. The StaxRip developer also ran tests and got the same results.
Maybe I’m misconfiguring something or misunderstanding a detail, since it worked in your example.
I’ll send you the error logs shortly. I’d really appreciate it if you could take a look!

.\NVEncC64.exe --avhw --qvbr 30 --codec av1 --output-depth 10 --colormatrix bt709 --colorprim bt709 --transfer bt709 -c av_libsvtav1 --avcodec-prms "preset=6,crf=30,svtav1-params=enable-variance-boost=1:variance-boost-strength=2" --crop 0,164,0,164 -i .\ZZZ6.mkv -o .\ZZZ6.mkv_temp\ZZZ6_NVEncC-Test1836_out.mkv

[rigaya]

I realized that it will need extra work with mkv output with av_libsvtav1, currently only works with mp4 output (but it'll be a problem as Staxrip seems to use mkv output). I currently have no idea what is causing this mkv output failure, I'll check if I can fix when I have time.

Also, you will need to remove --max-cll, --master-display (when used) as it cannot be used with -c av_libsvtav1 and switch to adding these like below.

--avcodec-prms "preset=6,crf=30,svtav1-params=enable-variance-boost=1:variance-boost-strength=2mastering-display=G(0.265,0.690)B(0.1500,0.0600)R(0.6800,0.3200)WP(0.3127,0.3290)L(1000.0,0.05),content-light=1000,300

[johnvick]

I’m not explicitly changing the values in the application, therefore not sure for each specific values. Instead, the driver is replacing various options for you when using --tune uhq to optimistic value. This should be what is expected which is described in NVIDIA documentation.

Does --tune uhq do away with the need for the quality preset e.g. --preset p7 ? My testing with AV1 and 265 using --tune uhq gives 330 fps at P1 and 100 fps at P7 - and I can't see a difference. Source videos are one hour 4 GB TV rips at 1080.

[rigaya]

I've fixed -c av_libsvtav1 to run fine with mkv output in NVEnc 9.07.

Also, I've added new option --check-preset-params in NVEnc 9.07 to allow show which parameter the driver changes by --codec, --preset and --tune.

You can check parameter which the driver sets, by for example, NVEncC64.exe --check-preset-params -c av1 --tune uhq -u P7 will show

#1: NVIDIA GeForce RTX 4080 (9728 cores, 2505 MHz)[PCIe4x16][581.42]
Codec: H.265/HEVC, Preset: quality, Tune: uhq
Changed NV_ENC_CONFIG fields:
  presetCfg.profileGUID: main
  presetCfg.gopLength: 250
  presetCfg.frameIntervalP: 6
  presetCfg.frameFieldMode: 1
  presetCfg.mvPrecision: Q-pel
  presetCfg.rcParams.rateControlMode: VBR
  presetCfg.rcParams.constQP.qpIntra: 25
  presetCfg.rcParams.constQP.qpInterP: 28
  presetCfg.rcParams.constQP.qpInterB: 31
  presetCfg.rcParams.enableLookahead: 1
  presetCfg.rcParams.lookaheadDepth: 25
  presetCfg.rcParams.lookaheadLevel: 3
  presetCfg.encodeCodecConfig.hevcConfig.maxCUSize: 32
  presetCfg.encodeCodecConfig.hevcConfig.chromaFormatIDC: 1
  presetCfg.encodeCodecConfig.hevcConfig.idrPeriod: 250
  presetCfg.encodeCodecConfig.hevcConfig.useBFramesAsRef: middle
  presetCfg.encodeCodecConfig.hevcConfig.tfLevel: 4
  presetCfg.encodeCodecConfig.hevcConfig.hevcVUIParameters.videoFormat: undef
  presetCfg.encodeCodecConfig.hevcConfig.hevcVUIParameters.colourPrimaries: undef
  presetCfg.encodeCodecConfig.hevcConfig.hevcVUIParameters.transferCharacteristics: undef
  presetCfg.encodeCodecConfig.hevcConfig.hevcVUIParameters.colourMatrix: undef

[jonrodrigos]

Hi rigaya, thanks a lot for your response and support!
As soon as I get some time, I'll run the tests with the changes you added.
I'll come back here to let you know how it goes.
Thanks again!