[embedder api] Vulkan External Resources

[jwinarske]

Use case

Per discussion on Discord - Use case gaps in order to switch to external embedder Vulkan backend

Vulkan external VkImage Support

1. GL interop. Same or different process. Source could be anything that inherently requires GL/GLES/EGL.
2. GStreamer pipeline output. Pipeline creates Vulkan objects directly via vulkanupload pipeline stage. Same or different process. Example pipeline which exposes VkImage to CPU:
   https://gist.github.com/jwinarske/dda0f7154a0bf34c3e7b6606907ad0f7
3. Present render target (VkImage) from (Unity, Filament, etc).
4. libcamera - source format is determined by the Camera sensor, and has to be determined at runtime.

External processes would have an array of VkImages available to present.

Proposal

The proposal is to adopt the OpenGL external Texture pattern for Vulkan. This would allow re-use of the TextureWidget.

GL Texture API:

FlutterEngineRegisterExternalTexture
FlutterEngineUnregisterExternalTexture
FlutterEngineMarkExternalTextureFrameAvailable

TextureFrameCallback would need to be updated to include VkImageInfo which describes the backing VkImage. Skia GrVkImageInfo is used to construct a GrBackendTexture.

[jonahwilliams]

@jwinarske is this something you are intending to work on yourself that you would like guidance on, or is this a feature request for the engine team?

[perkele1989]

This is something I'll likely need for day job in near future. Curious about progress on this, and if there's any way I could contribute.

[jonahwilliams]

The flutter engine/impeller teams have no plans to work on this in the near future. If someone contributed a working implementation (with tests) I think we could accept it?

[perkele1989]

Thank you for responding. may look into it going forward. for now leaning to do a CPU roundtrip and see if its performant enough. Seems like most of the work/time investment would involve setting up a development environment. The actual codebase seems very ready for it already to accept external Vulkan VkImages/VkImageViews via something like VK_KHR_external_memory, the same way it can do so from Java.