pixstream in VR mode The frame rate is very choppy.

[dvbhnijo45156213]

In UE 5.4.4, I created a virtual reality project and used a signaling server for streaming. When not in VR mode, the streaming bandwidth is around 90 Mbps. However, when entering VR mode, the frame rate becomes very choppy and the streaming bandwidth drops below 5 Mbps.My VR device is the VIVE Cosmos.Perhaps optimizations could be gleaned from how three.js handles panoramic VR videos, but it appears that three.js has issues when it comes to rendering video streams to VR devices. The videos in three.js are panoramic rather than simple videos. It is currently unclear whether the problem lies with the UE program overloading the rendering for the left and right eyes (with issues in the traffic statistics algorithm under the VR model) or whether the front-end is too inefficient at updating video textures?

https://threejs.org/examples/?q=vr#webxr_vr_video
Could we create a separate Discord server to facilitate discussions?

[dvbhnijo45156213]

I think this is a performance issue. The JavaScript main thread is responsible for decoding and encoding video, but the XR session severely interferes with the main thread's performance. The main thread accepts video more slowly, and at the same time, the XR session captures video frames and renders the context to VR, which also consumes a significant amount of performance.

[dvbhnijo45156213]

At the same time, the XR session cannot be moved to a Web Worker because WebXR is not available in Web Workers. That's really frustrating.

[dvbhnijo45156213]

Currently, I'm using WebWorker to decode the image data from WebRTC streams to avoid performance interference caused by WebRTC and WebXR running on the main thread at the same time. The frame rate of the received frames can basically match the pushed frame rate, and the actual refresh rate in VR can reach above 72Hz. However, when I use a VR device, the image still feels choppy. I'm using Wi-Fi 5 and a Quest 3 device. How can I solve this problem?

[dvbhnijo45156213]

In pixel - streaming VR mode, I'm using the front - end WebXR - acquired device matrices of the left and right eyes to set the UE back - end program's camera angle. But unlike entering a pre - rendered scene, this method causes jerky transitions and a stuttering sensation. I tried interpolating the camera's rotation angle during headset rotation for smoother visuals, yet it made the camera turn sluggishly. Hence, this approach is unworkable. I think the viable solution is to perform back - end frame interpolation when the headset rotates rapidly.

[lukehb]

I think the viable solution is to perform back - end frame interpolation when the headset rotates rapidly.

I agree that is probably going to be the best solution to get a smooth, less jittery result.

Additionally, another solution that has been discussed around this issue is using panoramic/360 videos for the stream, which lets the user rotate freely in VR without the requirement to wait for UE to update the rotation/render/send a new frame on its side.

Obviously, the 360 video solution is still subject to latency/jitter when you have HMD translation so I personally think the frame/HMD interpolation/prediction method is the one that should attempted as this addresses the translation and rotation issues without needing to change much about what is sent. This technique is more or less what NVIDIA cloud XR uses from what I understand.

As for actually implementing this, it seems the best approaches from the literature is to train a small neural net to make predictions about the HMD transform given previous data. The trick in creating such a neural net will be collecting a sufficient amount of training data of real HMD movements. Such a dataset is not freely available from what I know, so that is the current line of thinking.

I have tried algorithmic predictors such as kalman filters, but the results are not robust enough to the variety of movements a user can perform in VR (at least in my testing).

[dvbhnijo45156213]

I suggest that the video stream during head rotation could be saved as a test, which I have already attempted. The resulting footage appears to be unsmooth, with noticeable jittering. This indicates that the streaming source itself is problematic. Initially, I speculated that the backend might be losing some data during position capture, as the position data isn't queued and consumed sequentially. However, this seems unlikely, as I've tested and found that the tick execution speed for obtaining positions is much faster than the frequency of updating positions in the receiving function. Another possibility could be that the frame capture function isn't capturing frames smoothly enough, though I haven't tested this yet. Based on personal experience, it seems feasible to determine whether the HMD is rotating and then perform frame interpolation accordingly. At the very least, it's essential to first ensure that the video stream source itself is smooth.

[lukehb]

At the very least, it's essential to first ensure that the video stream source itself is smooth.

To do this:

Video stream source smoothness (on the UE side) can be confirmed by launching your UE app using -PixelStreamingDebugDumpFrame, which will "Dumps frames from the encoder to a file on disk for debugging purposes".

Encoded frames are dumped in a file the project's /Saved/ directory will a name like encoded_frame.raw

These frames can then be viewed using ffmpeg -i "encoded_frame.raw" "frames/out-%04d.jpg" or similar ffmpeg commands.

[dvbhnijo45156213]

Are you sure that the video source is smooth by itself? I mean, have you conducted detailed tests to verify this when the head-mounted display (HMD) is turned quickly? I tested the Vive Commons device to watch a local smooth video and wrote logic to keep the video image always in the center of the field of view (the video plane is in the spatial coordinate system). However, when I turned my head, the image would jitter. This indicates that on some devices, the jitter is not related to the video source, but is caused by the device itself (the test video resolution is 2024×1024). Meanwhile, I used the Quest 3 to perform the same operation and turned my head, but the image did not jitter and was very smooth. This rules out the issue of differences between different devices. It also shows that as long as the video source itself is smooth, even when the head is turned, the image seen will be smooth. This test is based on a VR code example of watching a video on a wall in a room, which is modified from a web sample. In addition, to rule out the problem of spatial coordinate system writing and screen coordinate system writing, I made some changes to the UE pixel stream front-end code, added parameters in the shader to fix the rendering canvas to a certain place in space to simulate a fixed playback screen, and then observed the image on the screen. At this time, it can be seen that the image jitters when the head-mounted display is turned quickly. Therefore, I suspect that the video source is not captured smoothly when the head-mounted display is turned quickly.

[lukehb]

I used the Quest 3 to perform the same operation and turned my head, but the image did not jitter and was very smooth

I have also only done verification of Quest 2 and Quest 3 using non-Pixel Streaming video sources and Pixel Streaming video sources. The interesting thing is I see smooth Pixel Streaming video until I enter "immersive" XR mode on the Meta Quest web browser on the Pixel Streaming page. When I view the video on the webpage normally (not in immersive XR mode) it looks perfectly smooth to me (however, admittedly jitter would be hard to detect when the video is not locked to direct view and is instead static on the page).

Note: By "immersive" XR mode I mean clicking the enter VR icon on the Pixel Streaming page.

Are you sure that the video source is smooth by itself? I mean, have you conducted detailed tests to verify this when the head-mounted display (HMD) is turned quickly?

I have done some testing, but if you want to verify this with a "detailed" test, the answer I gave in #591 (comment) should help you do so, but to be clear, I have not done this particular test I shared with you in that comment. I look forward to your results.

Or said very clearly, to avoid any doubt:

"Are you sure that the video source is smooth by itself?"

No.

[dvbhnijo45156213]

Yes, I agree. To verify this issue, I first referred to WebXR Samples. I downloaded its source code and used the bbb-sunflower-540p2-1min.webm video as a local video resource. I modified this example by removing the wall and keeping the video always in the center of the view (updating the node’s position in real time). I compared the rendering performance of Vive and Quest, which caused jitter. This shows that a smooth VR experience can be achieved when the video source itself is clear.

Secondly, I rewrote the original code, changing the screen coordinates to world coordinates, so that the entire picture is fixed in the world space. This is similar to the static page, where the picture itself does not move. Then I turned my head to observe whether the picture was jittering. In this way, on the one hand, I was testing in VR mode, and on the other hand, I simulated the situation of a static page. Finally, I found that the picture was not smooth at this time.
The main changes in my code are as follows:

[dvbhnijo45156213]

I have tried algorithms like the Kalman filter for prediction, but the results are not robust enough for the various actions users can perform in VR (at least in my tests).
Yes, I also previously suspected it was due to motion position noise. I tried using similar algorithms to smooth positions in WebXR, but it indeed proved to be unfeasible.

[dvbhnijo45156213]

I used to record the frames in the front-end code using MediaRecorder. For example, I recorded for 1 minute to see if it was smooth, but the video was not smooth. You mentioned this method, and I just tried it but couldn’t find the raw file.Do you have a more detailed explanation for this method?