WebM Video Builder #2200

akre54 · 2022-06-21T16:43:27Z

Add a VideoBuilder class

The web runs primarily realtime and interactive but there are a number of cases where you might want the content in the canvas to run out of realtime to render to a video. A very fast or very slow draw call, e.g.

This builds on the work of the GifBuilder class and adds a bit more info to the encoding-rfc. Please let me know if I should make any edits.

cc @chrisgervang and visgl/hubble.gl#236

Open questions

Is it right to wrap the various APIs or should the client be responsible for passing in a buffer? I'm leaning towards ease-of-use for common cases, but the client can also pass in a buffer.
Should this use the frameBuffer instead of using readPixels? Should it use readPixelsToBuffer?

Alternatives considered

MediaRecorder + canvas.getStream - only realtime
WebMWriter - uses canvas.toDataUrl() which incurrs costly serialization
VideoFrame - not supported in FireFox

…s backgrounded though

… Also doc and test updates

akre54 · 2022-06-21T16:51:22Z

modules/video/src/video-builder.ts

+      source instanceof WebGL2RenderingContext
+    ) {
+      const gl = source;
+      // const {width, height} = gl.canvas;


Either one is fine with me. Preference?

Either should be fine if they're both the buffer size. I've assumed at this point const {width, height} = gl.canvas; was the way to do it, but I only have limited theoretical knowledge of how it differs from GL.VIEWPORT.

Do you know if GL.VIEWPORT is meaningfully different? The spec suggests they are synced on the application level.

Pulling the correct size has been a source of issues in the past due to subtle differences in how libs handle it. Luma.gl uses gl.canvas. Mapbox did something different. Maplibrae has diverged now I think. I tried to sort it out in this issue and rfc, but I wouldn't be surprised if it comes up again in some corner case.

I say go with what you know. I only know enough to flag it might not be straight forward.

Oh interesting! And good to know, thanks for doing the legwork here.

Seems like pulling from gl.canvas is the move. I had no idea this was such a hot button issue!

loaders.gl is generally WebGL agnostic - it doesn't depend on WebGL but works with typed arrays which are optimized for WebGL use.

Also loaders.gl doesn't use Node.js constructs like Buffer, preferring to use typed arrays in all interfaces. Simply mentioning Buffer tends to pull in a 50KB polyfill in browser bundles...

I would have liked it more if this function just accepted a typed array and if necessary we offered some utility function for getting hold of a typed array from different input types.

I think in ever case here this function reduces buffer to an Uint8Array, right?

If so, the Buffer block be switched to a block that only accepts source instanceof Uint8Array

Yeah so one way or another, we have to send an ArrayBuffer instead of Uint8Array due to the way Web Workers handle shared memory. I think it's a bit cleaner for the implementing side (one less helper function that is required for all calls) if we handle the conversion on the library level for canvas, gl, and Buffers, but I also understand more generally wanting to work exclusively in the land of typed data (and, to Ib's point, not needing to pull in a huge node polyfill).

Ib, what are you expecting for the common use case for this library? Would this helper live in Loaders? Luma?

chrisgervang · 2022-06-23T22:20:16Z

modules/video/test/video-builder.spec.js

+
+    async onFinalize({gl}) {
+      const videoDataUrl = await videoBuilder.finalize();
+      t.ok(videoDataUrl, 'finalize() returns WebM video URL');


What should the two videos look like? Encoding is a newer frontier so I'm not sure if we've established how to verify it worked in a unit. We don't need to test webm-wasm, but maybe there a way to verify the video isn't blank/undefined, and the resolution is as expected?

cc @ibgreen

Yeah my thought here was to use the SnapshotTestRunner and then visually compare a couple of the frames (say, first, middle, last?). What do you think?

That sounds solid. Very curious if compression is deterministic or flaky.

My understanding is that since ScreenshotTest is perceptual, ideally it would return the same thing even if the bytes are different (though the two ImageDatas should have the same data with the same parameters, and from what I can tell from libvpx - the lib web-wasm wraps - it looks like there's an option to pass a debug mode flag to make the output deterministic).

chrisgervang · 2022-06-23T23:15:50Z

modules/video/src/video-builder.ts

+      source instanceof WebGL2RenderingContext
+    ) {
+      const gl = source;
+      // const {width, height} = gl.canvas;


Either should be fine if they're both the buffer size. I've assumed at this point const {width, height} = gl.canvas; was the way to do it, but I only have limited theoretical knowledge of how it differs from GL.VIEWPORT.

Do you know if GL.VIEWPORT is meaningfully different? The spec suggests they are synced on the application level.

Pulling the correct size has been a source of issues in the past due to subtle differences in how libs handle it. Luma.gl uses gl.canvas. Mapbox did something different. Maplibrae has diverged now I think. I tried to sort it out in this issue and rfc, but I wouldn't be surprised if it comes up again in some corner case.

I say go with what you know. I only know enough to flag it might not be straight forward.

chrisgervang · 2022-06-23T23:38:03Z

Regarding the open questions,

Is it right to wrap the various APIs or should the client be responsible for passing in a buffer? I'm leaning towards ease-of-use for common cases, but the client can also pass in a buffer.
- I think this is fine since a buffer is one of the supported options, and this removes complexity for the client.
Should this use the frameBuffer instead of using readPixels? Should it use readPixelsToBuffer?
- Is readPixelsToBuffer from luma? I had to google it.
- I would think handling a custom frameBuffer input is another branch of logic.
- Is readPixels better than canvas.toBlob since it's more direct?

modules/video/docs/api-reference/video-builder.md

ibgreen

Nice addition! Sorry for delay, I am sure I reviewed this a few days ago but somehow my comments didn't make it in.

modules/video/docs/api-reference/video-builder.md

ibgreen · 2022-06-24T16:18:48Z

modules/video/src/video-builder.ts

+}
+
+export default class VideoBuilder {
+  worker: Worker = new Worker('node_modules/webm-wasm/dist/webm-worker.js');


We typically include such workers in our own dist, rather than import them from installed packages. Given that it is apache licensed there are no concerns with copying it, especially if we give some attribution.

The libs folder in each module is the place to put such "binaries". There are a bunch of modules that do this.

The downside is we don't automatically get access to patches but I think it may cause problems for users if we depend on an npm install tree.

Done. It seems like they don't update all that regularly, so we should be fine for the time being.

That said, I'm having trouble with the webpack config here... doesn't seem to be finding the worker code when I try to import (I see there's a rule for test: /\.worker\.js$/, but it doesn't seem to be working). From what I can tell from the parquet-wasm and draco loader, it looks like they're loaded from third party CDNs. Any pointers on what to do here?

modules/video/docs/api-reference/video-builder.md

modules/video/src/video-builder.ts

ibgreen · 2022-06-24T16:23:01Z

modules/video/src/video-builder.ts

+      source instanceof WebGL2RenderingContext
+    ) {
+      const gl = source;
+      // const {width, height} = gl.canvas;


loaders.gl is generally WebGL agnostic - it doesn't depend on WebGL but works with typed arrays which are optimized for WebGL use.

Also loaders.gl doesn't use Node.js constructs like Buffer, preferring to use typed arrays in all interfaces. Simply mentioning Buffer tends to pull in a 50KB polyfill in browser bundles...

ibgreen · 2022-06-24T16:24:12Z

modules/video/src/video-builder.ts

+      source instanceof WebGL2RenderingContext
+    ) {
+      const gl = source;
+      // const {width, height} = gl.canvas;


I would have liked it more if this function just accepted a typed array and if necessary we offered some utility function for getting hold of a typed array from different input types.

akre54

Updated with feedback. Worker URL situation is still a little wonky, would love some help or pointers here.

akre54 · 2022-06-23T23:30:08Z

modules/video/src/video-builder.ts

+      source instanceof WebGL2RenderingContext
+    ) {
+      const gl = source;
+      // const {width, height} = gl.canvas;


Oh interesting! And good to know, thanks for doing the legwork here.

Seems like pulling from gl.canvas is the move. I had no idea this was such a hot button issue!

akre54 · 2022-06-28T00:58:29Z

modules/video/src/video-builder.ts

+      source instanceof WebGL2RenderingContext
+    ) {
+      const gl = source;
+      // const {width, height} = gl.canvas;


Yeah so one way or another, we have to send an ArrayBuffer instead of Uint8Array due to the way Web Workers handle shared memory. I think it's a bit cleaner for the implementing side (one less helper function that is required for all calls) if we handle the conversion on the library level for canvas, gl, and Buffers, but I also understand more generally wanting to work exclusively in the land of typed data (and, to Ib's point, not needing to pull in a huge node polyfill).

Ib, what are you expecting for the common use case for this library? Would this helper live in Loaders? Luma?

akre54 · 2022-06-28T01:03:08Z

modules/video/test/video-builder.spec.js

+
+    async onFinalize({gl}) {
+      const videoDataUrl = await videoBuilder.finalize();
+      t.ok(videoDataUrl, 'finalize() returns WebM video URL');


My understanding is that since ScreenshotTest is perceptual, ideally it would return the same thing even if the bytes are different (though the two ImageDatas should have the same data with the same parameters, and from what I can tell from libvpx - the lib web-wasm wraps - it looks like there's an option to pass a debug mode flag to make the output deterministic).

modules/video/docs/api-reference/video-builder.md

akre54 · 2022-06-28T03:03:39Z

modules/video/src/video-builder.ts

+}
+
+export default class VideoBuilder {
+  worker: Worker = new Worker('node_modules/webm-wasm/dist/webm-worker.js');


Done. It seems like they don't update all that regularly, so we should be fine for the time being.

That said, I'm having trouble with the webpack config here... doesn't seem to be finding the worker code when I try to import (I see there's a rule for test: /\.worker\.js$/, but it doesn't seem to be working). From what I can tell from the parquet-wasm and draco loader, it looks like they're loaded from third party CDNs. Any pointers on what to do here?

modules/video/docs/api-reference/video-builder.md

modules/video/src/video-builder.ts

chrisgervang · 2022-07-05T23:09:36Z

modules/video/src/webm-video-builder.ts

+  realtime: false
+};
+
+type VideoBuilderOptions = Partial<typeof VIDEO_BUILDER_OPTIONS>;


WebMVideoBuilderOptions

chrisgervang · 2022-07-05T23:09:53Z

modules/video/src/webm-video-builder.ts

@@ -0,0 +1,93 @@
+const VIDEO_BUILDER_OPTIONS = {


WEBM_VIDEO_BUILDER_OPTIONS

akre54 · 2022-07-06T22:13:01Z

Any thoughts about loading the worker file? Right now it depends on a specific hard coded path exposed through the server.

I couldn't get the Webpack loader to require the file (I gave up debugging and didn't look too deeply into the config).

chrisgervang · 2022-10-26T17:02:34Z

Any thoughts about loading the worker file? Right now it depends on a specific hard coded path exposed through the server.

I couldn't get the Webpack loader to require the file (I gave up debugging and didn't look too deeply into the config).

How does the rest of loaders handle this, @ibgreen? It's not clear to me if this will this cause any issues as-is for users?

akre54 added 3 commits June 15, 2022 21:56

initial test of VideoBuilder class

dd5d685

rewrite VideoBuilder to avoid the worker - might be better if this wa…

e25098b

…s backgrounded though

Get the buffer from the passed in context, still working out the API.…

00e3f89

… Also doc and test updates

akre54 commented Jun 21, 2022

View reviewed changes

chrisgervang reviewed Jun 23, 2022

View reviewed changes

ibgreen reviewed Jun 24, 2022

View reviewed changes

modules/video/docs/api-reference/video-builder.md Outdated Show resolved Hide resolved

ibgreen approved these changes Jun 24, 2022

View reviewed changes

Feedback

375462e

akre54 commented Jun 28, 2022

View reviewed changes

chrisgervang approved these changes Jul 5, 2022

View reviewed changes

chrisgervang changed the title ~~Video builder~~ WebM Video Builder Jul 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WebM Video Builder #2200

WebM Video Builder #2200

akre54 commented Jun 21, 2022

akre54 Jun 21, 2022

chrisgervang Jun 23, 2022

akre54 Jun 23, 2022

ibgreen Jun 24, 2022

ibgreen Jun 24, 2022

chrisgervang Jun 27, 2022 •

edited

akre54 Jun 28, 2022

chrisgervang Jun 23, 2022

akre54 Jun 23, 2022

chrisgervang Jun 24, 2022

akre54 Jun 28, 2022

chrisgervang Jun 23, 2022

chrisgervang commented Jun 23, 2022

ibgreen left a comment

ibgreen Jun 24, 2022

akre54 Jun 28, 2022

ibgreen Jun 24, 2022

ibgreen Jun 24, 2022

akre54 left a comment

akre54 Jun 23, 2022

akre54 Jun 28, 2022

akre54 Jun 28, 2022

akre54 Jun 28, 2022

chrisgervang Jul 5, 2022

chrisgervang Jul 5, 2022

akre54 commented Jul 6, 2022

chrisgervang commented Oct 26, 2022

WebM Video Builder #2200

Are you sure you want to change the base?

WebM Video Builder #2200

Conversation

akre54 commented Jun 21, 2022

Add a VideoBuilder class

Open questions

Alternatives considered

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisgervang Jun 27, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisgervang commented Jun 23, 2022

ibgreen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akre54 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akre54 commented Jul 6, 2022

chrisgervang commented Oct 26, 2022

chrisgervang Jun 27, 2022 •

edited