[WebGL] Texture to tensor API #6853

Linchenn · 2022-09-24T00:11:24Z

This fixes #3937

Changes

Add a new method to the API: tf.tensor(values: WebGLData, shape, dtype) that allows users to create tensors from textures and WebGLData is an object of WebGLTexture, height, width and channels properties and the channels could be a non-empty subset of 'RGBA' in any order.

Reference:
#6848 (The current PR starts from this PR of Na.)
#4376

To see the logs from the Cloud Build CI, please join either our discussion or announcement mailing list.

This change is

qjia7

The current design is to extend and expose the webgl backend's capacity directly but not expose it through tfjs-core. So you have come to an agreement to do it like this?
Two questions in my mind:

Usually, users check https://js.tensorflow.org/api/latest/ to get all latest APIs. How should we let users know the backend directly exposed APIs?
dataToGPU has been exposed by tfjs-core. It's wired that createTensorFromTexture is not like this way. And you also mentioned that the user can get texture by tf.backend().getTexture(a.dataId). So dataToGPU is duplicated with getTexture from functionality?

tfjs-backend-webgl/src/backend_webgl.ts

tfjs-backend-webgl/src/webgl.ts

qjia7 · 2022-09-26T01:58:52Z

tfjs-backend-webgl/src/webgl.ts

+ *  @param texture The WebGL texture to create a tensor from. The texture must
+ * be unpacked - each texel should only store a single value. The flattened
+ * values of the tensor will be read from left to right, and top to bottom. The
+ * texture can have empty texels at the end, but the values must be written


The empty texels mean zero texels, right? When the texel size is larger than shape size, the extra texels will be filled with zero.
And it's confused to say that all empty texels must be contiguous.

qjia7 · 2022-09-26T02:03:48Z

tfjs-backend-webgl/src/webgl.ts

+ * densely - in other words, all empty texels must be contiguous.
+ *  @param shape The logical shape of the texture.
+ *  @param dtype The dtype of the tensor to be created.
+ *  @param texShapeRC The physical dimensions of the texture expressed as [rows,


Is it better to use the same name with gl.texImage2D, like width/height? So you can say The width of the texture provided to gl.texImage2D during texture creation.
The height of the texture provided to gl.texImage2D during texture creation.. Similar to the other params (internalFormat -> internalformat, textureFormat -> format, textureType -> type)?

qjia7 · 2022-09-26T02:17:54Z

tfjs-backend-webgl/src/webgl.ts

+  const texConfig = backend.gpgpu.textureConfig;
+  let params: gpgpu_util.TextureCreationParams;
+
+  const isPacked = textureFormat === gl.RGBA;


Since you require that the texel of input texture must be one value, it's not possible that textureFormat is equal to gl.RGBA?

In fact, I prefer you support both unpacked texture and the densely rgba texture as the input.

Done. Now, we support all non-empty subsequence of 'RGBA'.

lina128

Hi Lin, thank you for your hard work on this important feature! Sorry I probably wasn't clear when explaining the design of this method. I meant to say that let's follow Jiajia's Life of a Tensor design of the API, please refer to the doc for detail. Basically, the method should be in tensor.ts's constructor function. async tensor(...), it's at tfjs-core level. Also, we should support this two formats, R32F and RGBA, for R32, just pass through, for RGBA, do dense to unpacked using encodeMatrix. Thank you so much!!! I think only this two changes, plus one more test, then I'm all good. Also, can you also add the following reference to your PR description? Thanks. ref:#6848

Reviewable status: 0 of 1 approvals obtained (waiting on @Linchenn and @qjia7)

tfjs-backend-webgl/src/backend_webgl.ts line 181 at r1 (raw file):

  // Writes a new entry to the data store with a WebGL texture, and registers it
  // to the texture manager.

The doccomment is outdated, update?

Code quote:

  // Writes a new entry to the data store with a WebGL texture, and registers it
  // to the texture manager.

tfjs-backend-webgl/src/backend_webgl.ts line 219 at r1 (raw file):

    gpgpu_math.runProgram(
        this.gpgpu, binary, inputTensorData, outputTensorData,
        null /* customUniformValues */);

We can provide uniform values for texShape is useUniform flag is true, right?

Code quote:

/* customUniformValues */

tfjs-backend-webgl/src/backend_webgl_test.ts line 1331 at r1 (raw file):

    const texture = gl.createTexture();
    const tex2d = gl.TEXTURE_2D;
    const internalFormat = gl.RGBA;

Can we add a test that takes a texture with RGBA format and densely laid out? Basically, for values 0-11, only need a texture with 3 texels.

Code quote:

gl.RGBA

tfjs-backend-webgl/src/webgl.ts line 97 at r8 (raw file):

Previously, qjia7 (Jiajia Qin) wrote…

Why we don't accept the densely rgba texture?

Can we accept dense rgba texture? I think we want to support two texture formats: R32 and RGBA. And we check, if R32, then we just clone the texture, it's already unpacked, if RGBA, we run the encodeMatrix to convert the layout from dense to unpacked. Basically we output a tensor with unpacked texture (same behavior has creating a tensor with value on CPU, which will create an unpacked texture).

tfjs-backend-webgl/src/webgl.ts line 135 at r8 (raw file):

Previously, qjia7 (Jiajia Qin) wrote…

Since you require that the texel of input texture must be one value, it's not possible that textureFormat is equal to gl.RGBA?

In fact, I prefer you support both unpacked texture and the densely rgba texture as the input.

Agreed with Jiajia.

arpu · 2022-09-27T17:48:30Z

this is amazing! any code snipe for threejs ?

lina128

The implementation is so clean, thank you Lin! LGTM. Clean up test and debug code before merge.

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @Linchenn and @qjia7)

pyu10055

great work, thanks!

Reviewable status: complete! 2 of 1 approvals obtained (waiting on @lina128, @Linchenn, @pyu10055, and @qjia7)

tfjs-backend-webgl/src/backend_webgl.ts line 1320 at r28 (raw file):

      throw new Error(
          `The texture is invalid. Also, please make sure the texture and ` +
          `the TFJS WebGL backend are using same canvas. If you want to use ` +

using the same cavas

Linchenn

Reviewable status: complete! 2 of 1 approvals obtained (waiting on @lina128, @pyu10055, and @qjia7)

tfjs-backend-webgl/src/backend_webgl.ts line 1320 at r28 (raw file):

Previously, pyu10055 (Ping Yu) wrote…

using the same cavas

Done. Thanks!

pyu10055

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @lina128, @Linchenn, @pyu10055, and @qjia7)

tfjs-core/src/ops/tensor.ts line 89 at r29 (raw file):

 * // channels of Pixel1...
 *
 * // For postprocessing on the GPU, it's possible to retrieve the texture

probably they should use the dataToGPU API for post processing, not grabbing the texture directly (otherwise texture is in packed format)

Linchenn

Reviewable status: complete! 1 of 1 approvals obtained (waiting on @lina128, @pyu10055, and @qjia7)

tfjs-core/src/ops/tensor.ts line 89 at r29 (raw file):

Previously, pyu10055 (Ping Yu) wrote…

probably they should use the dataToGPU API for post processing, not grabbing the texture directly (otherwise texture is in packed format)

Great catch! Thanks!

pyu10055

Reviewed 1 of 7 files at r10, 1 of 1 files at r13, 2 of 3 files at r16, 5 of 9 files at r22, 1 of 5 files at r25, 1 of 1 files at r26, 1 of 3 files at r27, 1 of 2 files at r28, 1 of 1 files at r29, 1 of 1 files at r30, all commit messages.
Reviewable status: complete! 2 of 1 approvals obtained (waiting on @lina128, @pyu10055, and @qjia7)

qjia7

Thanks Lin. It looks great. Just some small comments.

qjia7 · 2022-10-08T06:41:10Z

tfjs-core/src/ops/tensor.ts

- *     they will be encoded as utf-8 and kept as `Uint8Array[]`.
+ *     or a flat array, or a `TypedArray`, or a `WebGLData` object. If the
+ * values are strings, they will be encoded as utf-8 and kept as `Uint8Array[]`.
+ * If the values is a `WebGLData` object, the texture must share the same


It's better to also declare that the internal texture format for the input texture must be floating point or normalized integer since we are using floating point sampler type in shader. Otherwise, if the users give a signed/unsigned integer texture, they will meet unknown issues.
See spec https://registry.khronos.org/OpenGL/specs/es/3.0/GLSL_ES_Specification_3.00.pdf 8.8 Texture Lookup Functions

Thank you for the reminder and the reference!

qjia7 · 2022-10-08T06:45:21Z

tfjs-core/src/tensor_util_env.ts

  let firstElem: typeof val = val;

  if (isTypedArray(val)) {
    return dtype === 'string' ? [] : [val.length];
  }
+  if (typeof val === 'object' && 'texture' in val) {
+    return [val.height, val.width];


Why not use return [val.height, val.width * val.channels.length()]? Otherwise, A portion of the data will be truncated.

Great catch, done.

Also, since we need to fill 'RGBA' as default if channels parameter is null. I moved this 'filling default' logics from backend to core, because to do this inferring shape in core, we also need to know the channels in core.

qjia7 · 2022-10-08T07:04:37Z

tfjs-backend-webgl/src/backend_webgl.ts

+  // to the texture manager.
+  writeTexture(
+      texture: WebGLTexture, shape: number[], dtype: DataType,
+      texHeight: number, texWidth: number, color: string): DataId {


nit: color -> channels to keep consistent?

qjia7 · 2022-10-08T07:09:33Z

tfjs-backend-webgl/src/backend_webgl.ts

+
+    const shapeAs3D = webgl_util.getShapeAs3D(shape);
+    const program =
+        new EncodeMatrixProgram(shapeAs3D, false /* isByteArray */, color);


Do we need to provide a special path for RGBA by calling EncodeMatrixPackedProgram? In this case, we can avoid once runWebGLProgram to pack the tensor (PackProgram) in many situations.

RGBA input's packing schema is dense packing, while our tfjs's packing schema is square packing. As a result, even though both are packed (using all RGBA channels), we may need to transform the values through EncodeMatrixPackedProgram.

qjia7 · 2022-10-09T00:55:38Z

tfjs-backend-webgl/src/backend_webgl_test.ts

+       const physicalShape: [number, number] = [width, height];
+       const a = tf.tensor({texture, height, width, channels: 'RGBA'});
+
+       expect(a.shape).toEqual(physicalShape);


Since you have changed the inferShape, here also needs to be updated.
expect(a.shape).toEqual([width, height * channels.length]);

Done. Thanks!

qjia7

LGTM after bots become green. Thanks.

PS: Please also update your PR description to align with your current changes.

qjia7 · 2022-10-09T01:34:17Z

cc @axinging You can continue the implementation in webgpu. Please refer to the doc of Life of a Tensor and the implementation here. Thanks.

lina128 and others added 5 commits September 22, 2022 14:44

texture to tensor.

1c11971

ini

73b2916

lint

de32e12

tune tests

7209e2a

Merge branch 'master' into lins

24baec9

Linchenn requested review from lina128 and qjia7 September 24, 2022 00:11

Linchenn added 5 commits September 23, 2022 23:36

tune style

b7ee83a

Merge branch 'lins' of https://github.com/Linchenn/tfjs into lins

ad8d3ea

tune style

205b1f8

tune test for f16 env

daaa585

tune style

c60a220

qjia7 reviewed Sep 26, 2022

View reviewed changes

lina128 reviewed Sep 26, 2022

View reviewed changes

Linchenn added 2 commits September 26, 2022 16:00

rename texShapeRC

8cc128d

expose func to core

50172ac

Linchenn and others added 9 commits October 2, 2022 15:47

move to core

6ae774d

Merge branch 'master' into lins

563bdae

Update engine.ts

011acd3

add color map

03c9dbc

add color shader

4e54da4

rollback

4ee1c48

upd docs

ced3aab

Update backend_webgl.ts

7081c9c

Update backend_webgl.ts

3952626

lina128 approved these changes Oct 3, 2022

View reviewed changes

Linchenn added 3 commits October 3, 2022 11:45

add tests

6c4d0e6

test f16

961d520

tune style

122b6ce

pyu10055 approved these changes Oct 5, 2022

View reviewed changes

Linchenn and others added 3 commits October 5, 2022 10:34

Merge branch 'master' into lins

a08123e

typo

6d85b60

Merge branch 'lins' of https://github.com/Linchenn/tfjs into lins

0cdc770

Linchenn requested a review from pyu10055 October 5, 2022 17:36

Linchenn commented Oct 5, 2022

View reviewed changes

Merge branch 'master' into lins

59d0698

lina128 approved these changes Oct 5, 2022

View reviewed changes

pyu10055 requested changes Oct 5, 2022

View reviewed changes

Linchenn and others added 3 commits October 5, 2022 13:56

dataToGPU

c39e740

Merge branch 'lins' of https://github.com/Linchenn/tfjs into lins

ba02b0f

Merge branch 'master' into lins

dad105b

Linchenn commented Oct 5, 2022

View reviewed changes

Linchenn requested a review from pyu10055 October 5, 2022 20:58

pyu10055 approved these changes Oct 5, 2022

View reviewed changes

qjia7 reviewed Oct 8, 2022

View reviewed changes

Linchenn and others added 2 commits October 8, 2022 11:18

Merge branch 'master' into lins

2a4c531

fix

6187006

Linchenn requested a review from qjia7 October 8, 2022 18:41

qjia7 reviewed Oct 9, 2022

View reviewed changes

Linchenn added 2 commits October 8, 2022 18:16

fix

2ee87cb

typo

0bc4c44

qjia7 approved these changes Oct 9, 2022

View reviewed changes

Merge branch 'master' into lins

c3465e7

Linchenn merged commit 7281c8e into tensorflow:master Oct 9, 2022

Linchenn deleted the lins branch October 9, 2022 23:19

benbro mentioned this pull request Oct 10, 2022

New Texture to tensor API Volcomix/virtual-background#49

Closed

This was referenced Nov 10, 2022

Create tensor from GPUBuffer (GPUReadData) #6690

Closed

[webgpu] Create tensor from GPUBuffer #7034

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WebGL] Texture to tensor API #6853

[WebGL] Texture to tensor API #6853

Linchenn commented Sep 24, 2022 •

edited

qjia7 left a comment

qjia7 Sep 26, 2022

Linchenn Oct 3, 2022

qjia7 Sep 26, 2022

Linchenn Oct 3, 2022

qjia7 Sep 26, 2022

Linchenn Oct 3, 2022

lina128 left a comment

arpu commented Sep 27, 2022

lina128 left a comment

pyu10055 left a comment

Linchenn left a comment

pyu10055 left a comment

Linchenn left a comment

pyu10055 left a comment

qjia7 left a comment

qjia7 Oct 8, 2022

Linchenn Oct 8, 2022

qjia7 Oct 8, 2022

Linchenn Oct 8, 2022

Linchenn Oct 8, 2022 •

edited

qjia7 Oct 8, 2022

Linchenn Oct 8, 2022

qjia7 Oct 8, 2022

Linchenn Oct 8, 2022

qjia7 Oct 9, 2022

Linchenn Oct 9, 2022

qjia7 left a comment

qjia7 commented Oct 9, 2022

[WebGL] Texture to tensor API #6853

[WebGL] Texture to tensor API #6853

Conversation

Linchenn commented Sep 24, 2022 • edited

Changes

qjia7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lina128 left a comment

Choose a reason for hiding this comment

arpu commented Sep 27, 2022

lina128 left a comment

Choose a reason for hiding this comment

pyu10055 left a comment

Choose a reason for hiding this comment

Linchenn left a comment

Choose a reason for hiding this comment

pyu10055 left a comment

Choose a reason for hiding this comment

Linchenn left a comment

Choose a reason for hiding this comment

pyu10055 left a comment

Choose a reason for hiding this comment

qjia7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linchenn Oct 8, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qjia7 left a comment

Choose a reason for hiding this comment

qjia7 commented Oct 9, 2022

Linchenn commented Sep 24, 2022 •

edited

Linchenn Oct 8, 2022 •

edited