RAW Image access at Objective-c level

[timeinpixels]

Hello,

What would be the best way to access the image data at Objective-c level? Our codebase is in c++ and obj-c and I was wondering how should I go about implementing it properly.

Any hints would be greatly appreciated.

[michal-tomlein]

Hi,

The SDK doesn’t currently offer this kind of access. Can you describe your use case in more detail and explain why you need the image data?

Perhaps your use case is already covered or will be covered by future additions to the SDK on our roadmap.

[timeinpixels]

We would like to create an image analysis tool (port our existing desktop app). It heavily uses metal shaders for rendering and computation so ideally we would like to get each frame as Metal texture (or GPU buffer, IOSurface, etc).

[michal-tomlein]

Currently, the closest you can get to the image is by adding a CIFilter using the filters property of VideoViewController or VideoView. From there, you may be able to take the inputImage your filter receives and render it into a texture or buffer.

We’ll think about your use case and we’ll get back to you.

[timeinpixels]

Thank you. I will try that. Need to figure out the proper way of bridging swift to Objective-C, but should be possible.

[bradleyandrew]

@timeinpixels Did you manage to get anywhere with this approach? I did some initial testing and was able to get some data via a CIFilter 'inputImage' and 'outputImage' by passing in a local CIFilter that I control into the VideoViewController.

The input image is obtained like this:

if let filterInputImage = self.ciFilter.value(forKey: kCIInputImageKey) as? CIImage {
         print("ciFilter.inputImage: \(filterInputImage)")
}

And returns this:
ciFilter.inputImage: <CIImage: 0x1016fd850 extent [0 0 1920 1080]>

The approach using an output image would be slightly different, essentially you would create a CIFilter that does nothing, then tap into the 'ciFilter.outputImage' property and extract the MTLTexture from there. I used a Sepia Filter in my example but that would be swapped out in practice to something that does not manipulate the output image, below is the output:

ciFilter.outputImage: Optional(<CIImage: 0x1016fdbf0 extent [0 0 1920 1080]>
  colorkernel _sepia(s,amount=1) extent=[0 0 1920 1080] recipe=D10AD0EB
    MTLTexture 0x10166bc00 RGBA8Unorm extent=[0 0 1920 1080] recipe=FA0BA0DD
)

Another approach would be to use this function from VideoViewController: 'func renderImage() -> UIImage?'. You can extract a UIImage from the ScreenPort, then convert the UIImage to a CGImage and then the CGImage to a MTLTexture.

The problem with all of these approaches is that there is no callback. Ideally you would want to get these images every frame so you can extract the MTLTexture, put it through your pipeline and then present it in the UI. However there doesn't seem to be any way to get updates from the ScreenPortSDK when it receives new data.

I did look into the 'Device -> updatesPublisher' provided by the SDK but this only seems to update when a device connects/disconnects or the values of 'isNetworkConnectionAvailable' or 'isUSBConnectionAvailable' change.

A workaround is that you could take a user set frame rate and then run a recurring timer 25/30 times per second to create your own loop where you would extract the image either via the SDK Function or from within a CIFilter, but this wouldn't 100% be in sync with the ScreenPort Device and feels quite hacky.

Another thing I noticed in my testing is that no audio comes through via the ScreenPortSDK. Also there is no way to record video as you can only extract frames, of course as per the above workaround you could spin up your own video recording pipeline.

@michal-tomlein Is there anything in the pipeline that would address any of the above issues? In an ideal world there would be a VideoViewController Delegate that has callbacks with CMSampleBuffers for Video/Audio and ideally Metadata (Timecode and Record Flag). That way we could use the ScreenPort Device in a more flexible way, presumably this functionality already exists as it looks like it is used in the Official ScreenPort App so it'd just be about making it available via this SDK.

Thanks!

[michal-tomlein]

@bradleyandrew We are planning to introduce an API that will give you access to either compressed or uncompressed buffers.

We are leaning towards offering access to compressed frames (as received from ScreenPort) for the greatest flexibility, because decompressing the input for you means you have to do your own recompression, resulting in lower quality.

However, to be able to start recording at any time, this would mean you have to buffer all compressed frames from (and including) the last key frame, and correctly set the source time when writing to trim the start of the recording.

If you can share more about your use case and what you want to accomplish, that might help steer us in the right direction. Perhaps just exposing our own recorder (that already handles the above) as a higher-level API would work better for you.

[bradleyandrew]

@michal-tomlein Thank you for taking the time to get back to me. Great to know that this is still being worked on!

I couldn't find much information on how the ScreenPort works internally so it's tricky for me to weigh in on the compressed vs. uncompressed decision. I suspect what you mean though is that the video comes in via SDI through the ScreenPort and is then compressed for faster USB Transport to the device, is that right? What sort of compression?

I guess compressed buffers would be fine, though I am a not sure of the exact workflow you're suggesting in terms of buffering up until the last key frame? Couldn't video and audio buffers be sent via a delegate method each time a new buffer comes in so that the host app is always up to date? Then the app using the SDK can decide how to handle the buffers, either discard, display as a live video feed or record? Similar to how AVCaptureVideoDataOutputSampleBufferDelegate and AVCaptureAudioDataOutputSampleBufferDelegate work?

The use case is to utilise the ScreenPort as an SDI Capture Card on iPad, it gets used to display a live image, record video and capture frame grabs alongside real time video tools such as false color, LUTs, etc. I have had a number of crew members request ScreenPort support as they would like to use the hardware they already own rather than purchasing a new box for SDI Input.

Exposing your existing recorder wouldn't be ideal for my use case as I already have my own recording pipeline in play using AVAssetWriter and wouldn't want to maintain two capture pathways, one for ScreenPort and one for everything else. It would be much more streamlined if it slotted in with the existing workflow.

Exposing the existing recorder may be a useful option for people who wish to capture video without building their own pipeline, though I suspect most devs looking to integrate the ScreenPortSDK would already have established workflows in their apps.

[michal-tomlein]

@bradleyandrew ScreenPort uses H.264 or HEVC (it is user-configurable in the ScreenPort app) for video compression.

The problem is that with these codecs, you can’t start recording on a non-key frame. Typically you get one key frame per second, the rest are non-key frames. If you want to start recording at frame 10, you have to give AVAssetWriter the previous 9 frames as well, and specify the time of the 10th frame as the source time (see my previous comment), so that on playback, the first 9 frames are only used by the decoder to decode the 10th frame, but not played.

Thanks for the info about your use case. Sounds like you’ll be decompressing and recompressing anyway. This can still produce good results if the bitrate coming from ScreenPort is high enough.

[bradleyandrew]

@michal-tomlein Thank you for that clarification. Currently my implementation uses AVAssetWriter and the CMSampleBuffers are passed in for audio/video as they come in from the delegate method so recording can happen at anytime without managing keyframes as you describe. But given that the output from the ScreenPort is H.264 or HEVC and may come in that form via the updated API sounds like it may need to be handled differently.

Did you have any sort of ETA in terms of the rollout of the updated API?