✨ Support depth data streaming

[mrousavy]

What feature or enhancement are you suggesting?

It would be great to have depth data streaming support in VisionCamera Frame Processor plugins.
Something like this:

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const depthData = frame.depth
  if (depth != null) {
    console.log(`Depth data: ${depth.width} x ${depth.height}`)
    const data = depth.toArrayBuffer() // <-- depth is also a `Frame`!
  }
}, [])

...assuming the CameraDevice supports depth data streaming (supportsDepthData)

What Platforms whould this feature/enhancement affect?

iOS, Android

Alternatives/Workarounds

Currently I need to create a custom fork of VisionCamera to add depth data streaming and synchronization (AVCaptureSynchronizedDepthData ) to it.

Also, @thomas-coldwell created a public PR for this here: #745

..but it is quite old already and would need an update.

On Android, there is no such thing as depth data streaming I think.

Additional information

• I agree to follow this project's Code of Conduct
• I searched for similar feature requests in this repository and found none.

[thomas-coldwell]

Hey 👋 I have an updated depth implementation in a Vision Camera fork so I can add this here.

One important API consideration though is that the depth map returned from the LiDAR or TrueDepth camera on iOS is just a CVPixelBuffer - no CMSampleBuffer container. I've tried to create one but it never creates successfully due to there not being a compatible format description for the CMSampleBuffer for the depth pixel format. This means it can't be wrapped in a Frame HostObject itself and instead I just have a depthMap on the native Frame class and report a hasDepthMap on the Frame HostObject. Maybe there is a better approach here e.g. an ImageBuffer HostObject that can wrap the CVPixelBuffer and expose all of its properties in a similar way to the Frame setup with a CMSampleBuffer - wdyt?

[mrousavy]

Yup, good idea - a minimal Frame instance that just holds pixels. I can reshape the API for that

[gianfelipe93]

any updates on this?

[lucasjohnston]

Hey @thomas-coldwell could you link your updated fork? 🙇

[harrigee]

@thomas-coldwell A link to the updated fork would be great :)