feat(WIP): support stream trailer

[climba03003]

Support stream trailer trail PR.
I open this PR in WIP for collecting feedback. For example, is there any better solution?

Checklist

• run npm run test and npm run benchmark
• tests and/or benchmarks are included
• documentation is changed or added
• commit message and code follows the Developer's Certification of Origin
  and the Code of conduct

[mcollina]

The change that makes trailer support promises and callbacks is very good. However, I don't think the sendStreamTrailer change is correct due the memory spike.

A memory friendly solution is to use https://github.com/mcollina/cloneable-readable to create a clone of the original stream and pipe one as a response and another to the trailer generation logic. This would also imply that we must call the trailer function immediately and not at the end of the response. Unfortunately, this would be a breaking change.

The good news is that once reply.trailer can be async, the stream support can be implemented in userland on top of it: create a clone of the stream in the main route handler and use that to compute the trailer value. This could be a good fit for a plugin:

const newStream = reply.wrapStreamWithTrailers(stream, {
  Etag: function (reply, payload, cb) {
    // fill in
    setImmediate(cb, null, '42')
  }
})
return newStream

[mcollina]

Unfortunately, this is the breaking change I mentioned.

[climba03003]

New issue comes, I have no way to know payload is ended when I use payload.pipe(res, { end: false }) in sendStream

[climba03003]

is there good choice without using .filter?
The previous implementation using a single variables will count wrong and have racing condition.

cc @Uzlopak maybe you will have some idea?

[mcollina]

You can implement the same with a for loop

[climba03003]

I am closing this PR since I don't have more time spent on it.
The major issue left in here is quite a critical one.

1. When .pipe(stream, { end: false }) is used. The trailer handler will never know when the stream is ended.
2. If we do not use clonable-readable, it face the memory issue.

I can give a hand if anyone would like to pick it up. Just ping me.

[mcollina]

We have quite a few months before we can v5. Maybe you can revisit this in a few months?

[climba03003]

Maybe you can revisit this in a few months?

Yes.
The major issue is stuck in trailer handler have no way to know when the stream is ended.
I am not a stream expert and it seems a correct behavior in node since we use .pipe(stream, { end: false }).
When the user cannot attach .on('end', () => {}) to signal the callback is done. We are open in never ended request.

One solution is a timeout to check whether new content comes. But it is a hacky solution that I don't want to implement.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.