Send raw bytes through data channel

[Ozzadar]

Component your feature request relates to

• Signalling server
• Frontend library
• Frontend UI library
• Matchmaker
• SFU
• This repo

Problem your feature solves? Please describe.
Currently, the only way I see to send arbitrary data through the data channel is to run everything through emitUIInteraction which runs everything throughJSON.stringify(). This makes sending arbitrary bytes through rather annoying.

Describe the solution you'd like

I'd like a function that sends a byte array ( Uint8Array.?) through the data channel without any processing.

    public emitBytes(bytes: Uint8Array) {
        if (!this._webRtcController.videoPlayer.isVideoReady()) {
            return false;
        }
        this._webRtcController.emitBytes(bytes);
        return true;
    }

Additional context

[lukehb]

Currently, the only way I see to send arbitrary data through the data channel is to run everything through emitUIInteraction which runs everything throughJSON.stringify(). This makes sending arbitrary bytes through rather annoying.

This is incorrect.

There is already an existing mechanism to send whatever payload you like over the datachannel - it is the same mechanism Pixel Streaming itself uses to send all its messages. While it is not documented, it should be quite stable in terms of APIs used.

To use It involves both sides - the frontend library and the Pixel Streaming plugin itself.

On the frontend side:

1. Use StreamMessageController, accessed via the PixelStreaming class, pixelstreaming._webRtcController.streamMessageController
2. Create a new message for the payload you want to send:

StreamMessageController.toStreamerMessages.set('MyCustomMessage', {
           id: 137, // Set an id that is not already used by Pixel Streaming, you can study this class to see which are used
           structure: ['uint8'] // Define the structure of your payload
       });

3. Send your new message using the SendMessageController, accessed via pixelstreaming._webRtcController.sendMessageController:

SendMessageController.sendMessageToStreamer("MyCustomMessage", MyPayload)

On the Pixel Streaming plugin side:

1. Use the IPixelStreamingInputHandler::RegisterMessageHandler function, accessed via

IPixelStreamingModule& Module = IPixelStreamingModule.Get();
TWeakPtr<IPixelStreamingInputHandler> InputHandler = Module.FindStreamer(Module.GetDefaultStreamerID())->GetInputHandler();

2. Bind a message handler on the UE side:

InputHandler->RegisterMessageHandler("MyCustomMessage", [](FString SourceId, FMemoryReader Ar){
    TPayloadOneParam<uint8> Payload(Ar);
   // Do something with Payload.Param1
});

If you want to see more examples of message handling, see Plugins\Media\PixelStreaming\Source\PixelStreamingInput\Private\PixelStreamingInputHandler.cpp

Additionally, if you need to send a payload of an unknown size, you can use or study the use of the string payload type.

Let me know if this is not suitable to your purposes, but this solution works today with no modification to UE, whereas the proposed method of emitBytes would require us to add new message on frontend/UE and you would have to wait until subsequent UE releases to actually use it, whereas what I have provided above works today, is extensible, and lets you define whatever payloads you want and have it handled all within the existing design of the Pixel Streaming system.

[Ozzadar]

This looks promising. Thanks, I'll give it a shot soon :)

[mio-moto]

Hi,

after some analyzing we found some inconsistencies in both frontend and the Unreal implemention, primarily that arbitrary sized commands can be sent from Unreal and received by the frontend library, but not in reverse.

This is because the frontend library needs an explicit layout that is being checked before sending, with the only dynamic size being string - which is insufficient if I'm trying to send blobs dressed as strings \x0 binary content.

This could be remedied by allowing

PixelStreamingInfrastructure/Frontend/library/src/UeInstanceMessage/SendMessageController.ts

Line 36 in 168e568

if (messageFormat === undefined) {

to bypass checks if the structure of a message is binary for example.

Unreal seems to handle a protocol registered like such just fine:

void UPixelStreamingBinaryChannel::RegisterChannels()
{
	FString Channel = "Custom Binary Message";
	FRegisterChannelReturnValue Result = RegisterIncomingMessage(Channel,
	                                                             // 13 bytes coming from the dgram
	                                                             {
		                                                             EPixelStreaming2MessageTypes::Undefined
	                                                             },
	                                                             [](FString SourceId, FMemoryReader MemoryReader)
	                                                             {
		                                                             int64 TotalSize = MemoryReader.TotalSize();
		                                                             UE_LOG(LogTemp, Warning,
		                                                                    TEXT(
			                                                                    "Received Array Buffer of size '%lld'"
		                                                                    ), TotalSize);
	                                                             });
}

FRegisterChannelReturnValue UPixelStreamingBinaryChannel::RegisterIncomingMessage(
	const FString& ChannelName,
	const TArray<EPixelStreaming2MessageTypes>& Layout, const IPixelStreaming2InputHandler::MessageHandlerFn& Handler)
{
	IPixelStreaming2Module& Module = IPixelStreaming2Module::Get();

	TSharedPtr<IPixelStreaming2Streamer> Streamer = Module.FindStreamer(Module.GetDefaultStreamerID());
	if (!Streamer.IsValid())
	{
		return {false, 0};
	}

	TSharedPtr<IPixelStreaming2InputHandler> InputHandler = Streamer->GetInputHandler().Pin();
	if (!InputHandler.IsValid())
	{
		return {false, 0};
	}

	// select the protocol direction
	TSharedPtr<IPixelStreaming2DataProtocol> Protocol = InputHandler->GetToStreamerProtocol();
	// this only registers that there exists a channel
	Protocol->Add(ChannelName, Layout);
	// however, to know the ID we will need to pull out the registered message
	TSharedPtr<IPixelStreaming2InputMessage> Message = Protocol->Find(ChannelName);
	if (!Message.IsValid())
	{
		UE_LOG(LogTemp, Error, TEXT("Failed to register Message Channel '%s' into the streaming protocol"),
		       *ChannelName)
		return {false, 0};
	}
	
	// to finally receive messages, the handler has to be registered by channel name, otherwise Unreal will drop the messages
	InputHandler->RegisterMessageHandler(ChannelName, Handler);
	return {true, Message->GetID()};
}

in the meanwhile, I've gone ahead an just monkey-patched the SendMessageController:

import { Logger } from '@epicgames-ps/lib-pixelstreamingfrontend-ue5.6'
import type { SendMessageController } from '@epicgames-ps/lib-pixelstreamingfrontend-ue5.6/dist/types/UeInstanceMessage/SendMessageController'

const patchSign = 'FF-MonkeyPatched-Func' as const
type PossibleSendMessageToStreamerFunc = SendMessageController['sendMessageToStreamer'] & { __patchSign?: typeof patchSign }

export const patchSendMessageController = (messageController: SendMessageController) => {
  const oldSendMessageToStreamer = messageController.sendMessageToStreamer
  const possiblyPatched = oldSendMessageToStreamer as PossibleSendMessageToStreamerFunc
  if (possiblyPatched.__patchSign && possiblyPatched.__patchSign === 'FF-MonkeyPatched-Func') {
    return
  }

  const newSendMessageToStreamer = (messageType: string, messageData?: Array<number | string>) => {
    const usedMessageData = messageData ?? []
    const toStreamerMessags = messageController.toStreamerMessagesMapProvider.toStreamerMessages
    const messageFormat = toStreamerMessags.get(messageType)

    if (messageFormat === undefined) {
      Logger.Error(
        `Attempted to send a message to the streamer with message type: ${messageType}, but the frontend hasn't been configured to send such a message. Check you've added the message type in your cpp`,
      )
      return
    }
    // if that is a raw message data type, use our patched sender
    if (messageFormat.structure && messageFormat.structure.length === 1 && messageFormat.structure[0] === 'raw' && usedMessageData) {
      if (usedMessageData.some((x) => typeof x === 'string' || x < 0 || x > 255)) {
        Logger.Error(`With message type: ${messageType} Expected an uint8 buffer, but either got an array of strings or values outside of [0...255]`)
        return
      }

      const arrayBuffer = new Uint8Array(usedMessageData as Array<number>)
      if (arrayBuffer.length >= 16384) {
        Logger.Error(`With message type: ${messageType} - uint8 Buffer is at length '${arrayBuffer.length}' and exceeds maximum size of 16383`)
      }
      const view = new DataView(new ArrayBuffer(arrayBuffer.length + 1))
      view.setUint8(0, messageFormat.id)
      for (let i = 0; i < arrayBuffer.length; i += 1) {
        view.setUint8(i + 1, arrayBuffer[i])
      }

      messageController.dataChannelSender.sendData(view.buffer)
      return
    }
    // in other cases, just use the regular message controller
    oldSendMessageToStreamer.call(messageController, messageType, usedMessageData)
  }
  newSendMessageToStreamer['__patch-sign'] = 'FF-MonkeyPatched-Func'
  messageController.sendMessageToStreamer = newSendMessageToStreamer
}

[lukehb]

@mio-moto If I am understanding your message correctly, the true limitation you are hitting is the frontend should also allow a dynamic sized message of bytes (not just string)?

From your message it seems the UE side is working fine for your purposes?