grpc-js: Allow per-channel request compression from the client and decompression from the server #1952

b0b3rt · 2021-10-25T05:21:40Z

This pull requests implements deflate and gzip compression on a per-channel basis (when instantiating a new client) and the corresponding decompression on the server side.

For the client, this is done through the addition of the grpc.default_compression_algorithm option to ChannelOptions, which is translated to the appropriate grpc-encoding header.

The server reads the grpc-encoding header and attempts the corresponding decompression, and fails with the status INTERNAL if the decompression fails, per the spec.

Support for this functionality has been requested in this issue.

I also recently found myself in a position where it'd be nice to have, so decided to give the whole open source thing a go :)

Other notes:

changes to package.json: when I first pulled down the repo and started running the setup scripts, they failed because a recent version of @types/node changed the type of Socket.localAddress from string to string?, causing compilation failures. If this change is actually a fix that reflects a pre-existing truth that localAddress could be missing/undefined at runtime, that could indicate that there's a fix required in Server.getChannelzSessionInfoGetter and Subchannel.getChannelzSocketInfo, but that'd be better addressed by a separate PR. The build similarly complained about not having @types/semver installed.
newly generated TestService files - these are used as fixtures for the tests I added to cover the new functionality, similar to how the files generated for channelz.proto are used in test-channels.ts.
usage of message flags - I'm not totally clear on the original motivation for using flags passed down through the message context to decide whether compression occurs (in compression-filter.ts). As far as I could determine, there isn't any way for a user to pass down flags when sending a message, so trying to use them to enable/disable compression would require strictly more work than just checking the ChannelOptions. Happy to modify this if I'm missing something.

…compression from the server

linux-foundation-easycla · 2021-10-25T05:21:44Z

The committers are authorized under a signed CLA.

✅ Robert M (7aa5b62, cec7e64)

murgatroid99

Thank you for working on this. It would be great to have this implemented.

However, some things need to be changed:

First, please do not put test code in src/ or proto/. test_service.proto already exists in test/fixtures/, and it shouldn't move from there. The corresponding generated code can go in test/generated/. This will require a new script line in package.json to generate it. channelz.proto and the corresponding generated code are used in src/channelz.ts, which is why they belong in proto/ and src/ respectively.

Second, now that the client can send using different compression methods, it should be handling the server's grpc-accept-encoding header. If the server does not accept the encoding the client said it was using, the client should disable compression on all messages it sends after receiving that header.

Third, the purpose of the NoCompress write flag is to disable compression on a per-message basis, overriding what the channel does overall. So, it still needs to be taken into account. Write flags can currently be set, at least for streaming calls. The user passes it here, in the encoding parameter of call.write, where it becomes context.flags. Then context.flags becomes writeObj.flags here.

Fourth, it would probably be helpful to export CompressionAlgorithms, either as currently written or as an enum, so that users can use it to more easily determine the value to pass to the new channel option.

packages/grpc-js/src/compression-filter.ts

murgatroid99 · 2021-10-25T16:20:00Z

packages/grpc-js/src/server-call.ts

@@ -32,6 +33,7 @@ import { StreamDecoder } from './stream-decoder';
 import { ObjectReadable, ObjectWritable } from './object-stream';
 import { ChannelOptions } from './channel-options';
 import * as logging from './logging';
+import { MetadataValue } from '.';


There's no real reason to represent the encoding as a MetadataValue through the code here. The header key doesn't end with -bin, so the value is just a string.

MetadataValue is just what's returned from calling .get when we initially pull out the encoding:
const encoding: MetadataValue | undefined = metadata.get('grpc-encoding')[0];

I can coerce it to a string if that'll reduce potential confusion, though.

We could narrow the actual return type in the Metadata definition using string literal types, but that would require a big jump in the minimum compiler version needed to use this library, so I'm not yet sure it's a good idea. In the meantime, I think this is a valid use of type coercion: we know that certain properties of the string guarantee certain narrower types, and the compiler just isn't aware of that.

So, in short, yes, please coerce the encoding to a string.

Sounds good, changed - I suppose it's also part of the spec that the grpc-encoding header needs to be a string value.

linux-foundation-easycla · 2021-10-26T01:46:23Z

The committers are authorized under a signed CLA.

✅ Robert M (7aa5b62, cec7e64, b2ebbee, 16f1356, 9d70f39, 959503e, af01007, d68d94a)

…compression, and fix test file generation

…server

b0b3rt · 2021-10-26T03:23:37Z

packages/grpc-js/src/channel-options.ts

@@ -36,6 +36,7 @@ export interface ChannelOptions {
  'grpc.enable_http_proxy'?: number;
  'grpc.http_connect_target'?: string;
  'grpc.http_connect_creds'?: string;
+  'grpc.default_compression_algorithm'?: number;


@murgatroid99 what do you think of changing this from number to keyof typeof CompressionAlgorithms (resulting in 0 | 1 | 2)? That'll probably be more legible to consumers of the library than just exporting CompressionAlgorithms.

That seems reasonable to me.

Ok, done. I also moved CompressionAlgorithms into its own file to avoid a circular dependency.

b0b3rt · 2021-10-26T03:31:42Z

Thank you for working on this. It would be great to have this implemented.

However, some things need to be changed:

First, please do not put test code in src/ or proto/. test_service.proto already exists in test/fixtures/, and it shouldn't move from there. The corresponding generated code can go in test/generated/. This will require a new script line in package.json to generate it. channelz.proto and the corresponding generated code are used in src/channelz.ts, which is why they belong in proto/ and src/ respectively.

Fixed :) - let me know if the new command is what you were thinking of, and whether it also needs to be called in prepare and not just pretest.

Second, now that the client can send using different compression methods, it should be handling the server's grpc-accept-encoding header. If the server does not accept the encoding the client said it was using, the client should disable compression on all messages it sends after receiving that header.

Implemented here. I'm not sure I see a way to do this only when the server returns UNIMPLEMENTED along with the grpc-accept-encoding header, but if there's a relatively clean way to avoid performing this check on every response received by the client let me know.

Third, the purpose of the NoCompress write flag is to disable compression on a per-message basis, overriding what the channel does overall. So, it still needs to be taken into account. Write flags can currently be set, at least for streaming calls. The user passes it here, in the encoding parameter of call.write, where it becomes context.flags. Then context.flags becomes writeObj.flags here.

Fixed, although it seems to be passed down through _write, not write (which afaict calls the base method from Http2CallStream, where it expects an encoding of type "ascii" | "utf8" | "utf-8" | ..., rather than being handled by gRPC-specific logic which understands WriteFlags). Let me know if I've missed something here.

Fourth, it would probably be helpful to export CompressionAlgorithms, either as currently written or as an enum, so that users can use it to more easily determine the value to pass to the new channel option.

Done, but also see my suggestion here.

packages/grpc-js/src/compression-filter.ts

murgatroid99 · 2021-10-26T17:09:38Z

Thank you for working on this. It would be great to have this implemented.
However, some things need to be changed:
First, please do not put test code in src/ or proto/. test_service.proto already exists in test/fixtures/, and it shouldn't move from there. The corresponding generated code can go in test/generated/. This will require a new script line in package.json to generate it. channelz.proto and the corresponding generated code are used in src/channelz.ts, which is why they belong in proto/ and src/ respectively.

Fixed :) - let me know if the new command is what you were thinking of, and whether it also needs to be called in prepare and not just pretest.

It looks fine. It doesn't need to be in prepare because we don't want the test code packed up or published.

Second, now that the client can send using different compression methods, it should be handling the server's grpc-accept-encoding header. If the server does not accept the encoding the client said it was using, the client should disable compression on all messages it sends after receiving that header.

Implemented here. I'm not sure I see a way to do this only when the server returns UNIMPLEMENTED along with the grpc-accept-encoding header, but if there's a relatively clean way to avoid performing this check on every response received by the client let me know.

Probably the best way to track this information across requests would be for the CompressionFilterFactory to own an object shared among all of the CompressionFilter instances that tracks the latest-received grpc-accept-encoding header. I don't think handling errors is necessary here. The filter can just do its best to only send messages that the server will accept.

Third, the purpose of the NoCompress write flag is to disable compression on a per-message basis, overriding what the channel does overall. So, it still needs to be taken into account. Write flags can currently be set, at least for streaming calls. The user passes it here, in the encoding parameter of call.write, where it becomes context.flags. Then context.flags becomes writeObj.flags here.

Fixed, although it seems to be passed down through _write, not write (which afaict calls the base method from Http2CallStream, where it expects an encoding of type "ascii" | "utf8" | "utf-8" | ..., rather than being handled by gRPC-specific logic which understands WriteFlags). Let me know if I've missed something here.

This is kind of a tangent, but these flags are getting passed in a couple of layers up from Http2CallStream, and _write is the internal override for write, so it will be called automatically when you call write. The types don't match, but if you forcibly pass in a number in the encoding argument of write, _write will see it and use it as the write flags.

…hm option is provided

b0b3rt · 2021-10-26T21:06:23Z

This is kind of a tangent, but these flags are getting passed in a couple of layers up from Http2CallStream, and _write is the internal override for write, so it will be called automatically when you call write. The types don't match, but if you forcibly pass in a number in the encoding argument of write, _write will see it and use it as the write flags.

Oh, I see. Since that's the case, I changed the test to call write with that flag, instead of _write.

I think this addresses all the outstanding feedback - my only open questions are around @types/node (is pinning the version ok, potentially pending a follow-up PR to address the type incompatibility introduced in more recent versions?) and adding @types/semver (not clear why it was refusing to build on my machine without it - I can try removing it for the PR's workflow tests, if you want).

packages/grpc-js/src/compression-filter.ts

murgatroid99 · 2021-10-26T21:19:53Z

I actually fixed the @types/node issue in #1953, so you can remove that change. The issue with @types/semver is the result of this repository's weird layout. You might as well just keep your change for that in.

Co-authored-by: Michael Lumish <mlumish@google.com>

…ovided

b0b3rt · 2021-10-26T21:27:21Z

I actually fixed the @types/node issue in #1953, so you can remove that change. The issue with @types/semver is the result of this repository's weird layout. You might as well just keep your change for that in.

Cool, reverted.

b0b3rt · 2021-10-28T16:41:14Z

@murgatroid99 I think I accidentally cancelled the test run by pushing a commit reverting the @types/node change. Are there any other changes needed before we can kick it off again?

murgatroid99

The overall structure and logic of this change look good now, but there are a few specific things that I think need to be changed.

murgatroid99 · 2021-10-29T17:04:40Z

packages/grpc-js/src/compression-filter.ts

@@ -167,10 +183,44 @@ function getCompressionHandler(compressionName: string): CompressionHandler {
 export class CompressionFilter extends BaseFilter implements Filter {
  private sendCompression: CompressionHandler = new IdentityHandler();
  private receiveCompression: CompressionHandler = new IdentityHandler();
+  private defaultCompressionAlgorithm: CompressionAlgorithm | undefined;


At this point this field would more accurately be called currentCompressionAlgorithm, since it's always set to the currently used algorithm, not the default.

Also, please remove | undefined from the type, and use "identity" instead of undefined as the default.

Makes sense.

murgatroid99 · 2021-10-29T17:04:43Z

packages/grpc-js/src/compression-filter.ts

  async sendMetadata(metadata: Promise<Metadata>): Promise<Metadata> {
    const headers: Metadata = await metadata;
    headers.set('grpc-accept-encoding', 'identity,deflate,gzip');
    headers.set('accept-encoding', 'identity');
+
+    if (this.defaultCompressionAlgorithm && ['deflate', 'gzip'].includes(this.defaultCompressionAlgorithm)) {


I don't like the explicit check for "deflate" and "gzip" here. It's effectively duplicating information in the CompressionAlgorithm type and CompressionAlgorithms object. It's perfectly valid to not have a check at all here, and to send "identity` when that is the value. Particularly with the other change, that would allow this code to have no check at all. Alternatively, I would be OK with explicitly excluding "identity", because that is the actual special case.

Yeah, good call, there's no reason to have this check anymore. I think explicitly excluding identity does make sense, if only to save on the bandwidth.

murgatroid99 · 2021-10-29T17:13:28Z

packages/grpc-js/src/compression-filter.ts

@@ -194,7 +257,7 @@ export class CompressionFilter extends BaseFilter implements Filter {
    const resolvedMessage: WriteObject = await message;
    const compress =


I think this assignment needs to be expanded out at this point. It's three conditionals and too many negatives, and I think the ordering is a little confusing. I think it would be better for the handler check to be the primary thing, and separate from the flags check. So, something like this:

if (this.sendCompression instanceof IdentityHandler) { compress = false; } else { compress = ((resolvedMessage.flags ?? 0) & WriteFlags.NoCompress) === 0; }

Yeah, it's a bit too much for a ternary expression. Split it out as suggested.

murgatroid99 · 2021-10-29T17:21:25Z

packages/grpc-js/src/server-call.ts

@@ -136,12 +137,13 @@ export class ServerReadableStreamImpl<RequestType, ResponseType>
  constructor(
    private call: Http2ServerCallStream<RequestType, ResponseType>,
    public metadata: Metadata,
-    public deserialize: Deserialize<RequestType>
+    public deserialize: Deserialize<RequestType>,
+    encoding?: string


As in the client-side filter code, throughout this file, encoding should be just a string, with "identity" used as the value indicating that there is no compression.

Yep, went ahead and made that consistent everywhere.

murgatroid99 · 2021-10-29T17:26:42Z

packages/grpc-js/src/server-call.ts

+        });
+      }
+
+      default:


There should be a separate "identity" case here that does not modify the message, and then the default case will be for unrecognized compression algorithms, and that should end the call with an UNIMPLEMENTED error.

packages/grpc-js/src/server-call.ts

murgatroid99 · 2021-11-01T15:49:25Z

packages/grpc-js/src/compression-algorithms.ts

+ *
+ */
+
+export const CompressionAlgorithms = {


What is the benefit of using const here instead of enum? I think an enum would be better for helping the user populate the channel option, because they could use CompressionAlgorithm.gzip, for example.

Generally I disprefer enums in Typescript because they're one of the few language features which aren't runtime-only, and they interact somewhat poorly with the rest of the type system, since they're somewhat poorly supported and the language designers seem to pushing people in the direction of using object literals (with as const) instead.

Those probably aren't huge concerns in this case; I don't see a way that making it an enum would frustrate a typical user of the library. For user convenience and consistency with the way other such options are handled, I'll change it to an enum.

Co-authored-by: Michael Lumish <mlumish@google.com>

…node into grpc-js_compression_support

murgatroid99 · 2021-11-02T16:23:45Z

packages/grpc-js/src/compression-filter.ts

+      const serverSupportedEncodings = serverSupportedEncodingsHeader.split(',');
+
+      if ((this.sendCompression instanceof DeflateHandler && !serverSupportedEncodings.includes('deflate'))
+          || (this.sendCompression instanceof GzipHandler && !serverSupportedEncodings.includes('gzip'))) {


I think this whole if statement can be reduced to if (!serverSupportedEncodings.includes(this.currentCompressionAlgorithm)), and it should also set this.currentCompressionAlgorithm.

Yep, simplified (and added).

murgatroid99 · 2021-11-02T16:23:52Z

packages/grpc-js/test/test-server.ts

+            `localhost:${assignedPort}`,
+            grpc.credentials.createInsecure(),
+            {
+              'grpc.default_compression_algorithm': 1


The test code should use the CompressionAlgorithms enum when setting this option, both for clarity and to demonstrate that the enum works as intended.

Good catch, thanks.

…ders

b0b3rt · 2021-11-05T02:00:46Z

Hm, looks like the tests are failing:

Error: ENOENT: no such file or directory, open 'test/fixtures/test_service.proto'
...
    at Object.loadSync (/tmpfs/src/github/grpc-node/packages/grpc-js/node_modules/@grpc/proto-loader/build/src/index.js:194:31)
    at Object.<anonymous> (/tmpfs/src/github/grpc-node/packages/grpc-js/test/test-server.ts:38:44)

which is
const loadedTestServiceProto = protoLoader.loadSync('test/fixtures/test_service.proto', {

Since this passes locally but not in the test runner, does the test runner have something other than grpc-node/packages/grpc-js as the root dir when running the test?

murgatroid99 · 2021-11-05T08:50:45Z

It looks like the problem is that you didn't update the includeDirs list when you moved the proto file. The path that you load needs to be an absolute path or relative to one in that list. The fixtures directory gets copied over to the build directory, so you can just load ${__dirname}/fixtures/test_service.proto without any include path.

b0b3rt · 2021-11-06T02:59:30Z

Got it, changed the path accordingly.

murgatroid99 · 2021-11-08T21:49:37Z

packages/grpc-js/src/server-call.ts

+  private getDecompressedMessage(message: Buffer, encoding: string) {
+    switch (encoding) {
+      case 'deflate': {
+        return new Promise<Buffer | undefined>((resolve, reject) => {
+          zlib.inflate(message.slice(5), (err, output) => {
+            if (err) {
+              this.sendError({
+                code: Status.INTERNAL,
+                details: `Received "grpc-encoding" header "${encoding}" but ${encoding} decompression failed`,
+              });
+              resolve();
+            } else {
+              const joined = Buffer.concat([message.slice(0, 5), output]);
+              resolve(joined);
+            }
+          });
+        });
+      }
+
+      case 'gzip': {
+        return new Promise<Buffer | undefined>((resolve, reject) => {
+          zlib.unzip(message.slice(5), (err, output) => {
+            if (err) {
+              this.sendError({
+                code: Status.INTERNAL,
+                details: `Received "grpc-encoding" header "${encoding}" but ${encoding} decompression failed`,
+              });
+              resolve();
+            } else {
+              const joined = Buffer.concat([message.slice(0, 5), output]);
+              resolve(joined);
+            }
+          });
+        });
+      }
+
+      case 'identity': {
+        return Promise.resolve(message);
+      }
+
+      default: {
+        this.sendError({
+          code: Status.UNIMPLEMENTED,
+          details: `Received message compressed with unsupported encoding "${encoding}"`,
+        });
+        return Promise.resolve();
+      }
+    }
+  }


I didn't notice this before, but this function has to actually remove the 5-byte chunk at the beginning of the message. That chunk contains the compressed flag and the length of the encoded message, which are both invalid and useless after the message has been decompressed. Then deserializeMessage also has to be modified to not slice off the first 5 bytes.

@murgatroid99 It does remove the 5-byte chunk before decompressing it; do you mean to avoid re-attaching it before returning it?

const joined = Buffer.concat([message.slice(0, 5), output]); resolve(joined);

Which would obviate the need to remove it again later:

deserializeMessage(bytes: Buffer) { // TODO(cjihrig): Call compression aware deserializeMessage(). const receivedMessage = bytes.slice(5); return this.handler.deserialize(receivedMessage); }

If that's what you meant, I'll make that change.

On a separate note, I'm not able to reproduce the test failures in Linux Tests. When I run the tests locally (on Ubuntu 20.04.1 via WSL2), the following two tests fail:

1) Name Resolver DNS Names Should resolve localhost properly: Error: Timeout of 4000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. 2) Name Resolver DNS Names Should default to port 443: Error: Timeout of 4000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

However, those tests were failing from the moment I pulled down the repo; I assumed it was a quirk of the local environment (or some minor dependency mismatch somewhere).

Yes, I meant to remove the 5 byte header from the output of that function, and the other code you referenced that removes it again.

Don't worry about those test failures. I see the same failures on WSL; it seems to be a quirk of how Windows or WSL resolves localhost.

Sounds good, pushed a commit that simplifies the removal of the "prefix".

murgatroid99

Thank you for your contribution, and for working with me to improve this PR.

desytech · 2021-12-27T22:36:02Z

@murgatroid99 is there any approximate release date for the compression support? Thanks

murgatroid99 · 2021-12-27T23:14:23Z

I will probably be able to publish this in early January after the holidays.

murgatroid99 · 2022-01-07T00:05:40Z

This has been published in version 1.5.0.

b0b3rt added 2 commits October 24, 2021 21:46

grpc-js: Allow per-channel request compression from the client and de…

7aa5b62

…compression from the server

cleanup test file

cec7e64

murgatroid99 requested changes Oct 25, 2021

View reviewed changes

revert mistaken accept-encoding client header change

b2ebbee

b0b3rt added 5 commits October 25, 2021 18:54

remove unused compression algorithm key value

16f1356

remove copied test proto file and modify npm command

9d70f39

rewrite CompressionAlgorithms to use numeric keys and export it

959503e

have client restore default sendCompression if server doesnt support …

af01007

…compression, and fix test file generation

re-enable NoCompress flag behavior and check Compressed Flag byte on …

d68d94a

…server

b0b3rt commented Oct 26, 2021

View reviewed changes

packages/grpc-js/src/compression-filter.ts Show resolved Hide resolved

b0b3rt added 4 commits October 26, 2021 13:44

change encoding type from MetadataValue to string

6d75ea1

separate compression algorithms to avoid circular dependency

21b09e2

log a warning when invalid value for grpc.default_compression_algorit…

e03c115

…hm option is provided

change test to use write instead of _write

7a31b4a

change year in new copyright heade

25a1806

murgatroid99 reviewed Oct 26, 2021

View reviewed changes

packages/grpc-js/src/compression-filter.ts Outdated Show resolved Hide resolved

murgatroid99 added the kokoro:run label Oct 26, 2021

kokoro-team removed the kokoro:run label Oct 26, 2021

b0b3rt and others added 3 commits October 26, 2021 14:21

Update error message for invalid default_compression_algorithm

d01ff79

Co-authored-by: Michael Lumish <mlumish@google.com>

only attempt to use custom compression filter when option value is pr…

6ba0081

…ovided

only attempt to use custom compression filter when option value is pr…

53a2347

…ovided

murgatroid99 requested changes Oct 29, 2021

View reviewed changes

default to identity header everywhere instead of leaving it undefined

9150bdf

murgatroid99 reviewed Nov 1, 2021

View reviewed changes

packages/grpc-js/src/server-call.ts Outdated Show resolved Hide resolved

murgatroid99 reviewed Nov 1, 2021

View reviewed changes

b0b3rt and others added 3 commits November 1, 2021 18:38

Update packages/grpc-js/src/server-call.ts

5c61a6a

Co-authored-by: Michael Lumish <mlumish@google.com>

change CompressionAlgorithms to an enum

dc9752a

Merge branch 'grpc-js_compression_support' of github.com:b0b3rt/grpc-…

deff9d4

…node into grpc-js_compression_support

murgatroid99 reviewed Nov 2, 2021

View reviewed changes

simplify compression filter handling of server supported encoding hea…

c4d7fab

…ders

murgatroid99 added the kokoro:run label Nov 3, 2021

kokoro-team removed the kokoro:run label Nov 3, 2021

fix path for loading test_service proto

96ae102

murgatroid99 added the kokoro:run label Nov 8, 2021

kokoro-team removed the kokoro:run label Nov 8, 2021

Merge branch 'master' into grpc-js_compression_support

9cdd36f

murgatroid99 added the kokoro:run label Nov 8, 2021

kokoro-team removed the kokoro:run label Nov 8, 2021

murgatroid99 reviewed Nov 8, 2021

View reviewed changes

simplify removal of compression prefix bytes

69428b0

murgatroid99 added the kokoro:run label Nov 10, 2021

kokoro-team removed the kokoro:run label Nov 10, 2021

murgatroid99 approved these changes Nov 15, 2021

View reviewed changes

murgatroid99 merged commit fe58061 into grpc:master Nov 15, 2021

		@@ -194,7 +257,7 @@ export class CompressionFilter extends BaseFilter implements Filter {
		const resolvedMessage: WriteObject = await message;
		const compress =

grpc-js: Allow per-channel request compression from the client and decompression from the server #1952

grpc-js: Allow per-channel request compression from the client and decompression from the server #1952

Conversation

b0b3rt commented Oct 25, 2021 • edited

linux-foundation-easycla bot commented Oct 25, 2021 • edited

murgatroid99 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Oct 26, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b0b3rt commented Oct 26, 2021

murgatroid99 commented Oct 26, 2021

b0b3rt commented Oct 26, 2021

murgatroid99 commented Oct 26, 2021

b0b3rt commented Oct 26, 2021

b0b3rt commented Oct 28, 2021

murgatroid99 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b0b3rt commented Nov 5, 2021

murgatroid99 commented Nov 5, 2021

b0b3rt commented Nov 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

murgatroid99 left a comment

Choose a reason for hiding this comment

desytech commented Dec 27, 2021

murgatroid99 commented Dec 27, 2021

murgatroid99 commented Jan 7, 2022

b0b3rt commented Oct 25, 2021 •

edited

linux-foundation-easycla bot commented Oct 25, 2021 •

edited

linux-foundation-easycla bot commented Oct 26, 2021 •

edited