I know this very much likely is a unity bug, but I'll report it here anyways.
Tested with an empty scene. If you start recording. You will notice your memory gets filled up quickly.
And even if you stop recording, the memory does not get released. This is true in both editor and build. Interestingly, if you are in the editor and stop playmode, the memory still does not get freed. Change scene does nothing, force GC collect or unload assets does nothing. In memory profiler, it seems total allocated momery shows huge amount of memory is allocated, but you don't see that in any thing in your game. Strange.
My suspect is GPUAsync request and getdata allocated persistant memory, and they never get disposed. I could be wrong. But my evidences point to it. It could be just unity and has nothing to do HDRP, but I lack testing on other SRPs.