chore(qwik-city/middleware): add TextEncoderStream polyfill and tests

[PatrickJS]

add _TextEncoderStream_polyfill and tests

[cloudflare-pages]

Deploying qwik-docs with    Cloudflare Pages

Latest commit:	849a365
Status:	✅  Deploy successful!
Preview URL:	https://54153751.qwik-8nx.pages.dev
Branch Preview URL:	https://feat-polyfill-textencoderstr.qwik-8nx.pages.dev

View logs

[netlify]

✅ Deploy Preview for qwik-insights ready!

Name	Link
🔨 Latest commit	849a365
🔍 Latest deploy log	https://app.netlify.com/sites/qwik-insights/deploys/6646a90c1d317900088cb1ff
😎 Deploy Preview	https://deploy-preview-6310--qwik-insights.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[PatrickJS]

@wmertens when you get time can you review

[octet-stream]

Saw this PR and decided to add a few bits after looking closer myself to TextEncoderStream. This implementation is shorter, and I believe it's fine to keep it as it is, but I have a few thoughts from what I understand:

1. The TextEncoderStream class is not an extension of the TransformStream (you can see that from the documentation on MDN here), but it has same generic interface with readable and writable properties (see the spec here). I'm not sure if we are trying to achieve the most compatibility, but you can refer to Deno's and Node.js' implementations if that's the case.
2. There are properties that I don't see on this class either. I left more comments on the specifics :)

[PatrickJS]

yeah the props are there only to match the previous polyfill. so everything is just-in-case

[octet-stream]

Oh, sure. But anyway here's TypeScript flavored version of TextEncoderStream polyfill based on Node.js' implementation, just in case:

/**
 * TextEncoderStream polyfill based on Node.js' implementation https://github.com/nodejs/node/blob/3f3226c8e363a5f06c1e6a37abd59b6b8c1923f1/lib/internal/webstreams/encoding.js#L38-L119 (MIT License)
 */
export class TextEncoderStream {
  #pendingHighSurrogate: string | null = null

  #handle = new TextEncoder()

  #transform = new TransformStream<string, Uint8Array>({
    transform: (chunk, controller) => {
      // https://encoding.spec.whatwg.org/#encode-and-enqueue-a-chunk
      chunk = String(chunk)

      let finalChunk = ""
      for (const item of chunk) {
        const codeUnit = item.charCodeAt(0)
        if (this.#pendingHighSurrogate !== null) {
          const highSurrogate = this.#pendingHighSurrogate

          this.#pendingHighSurrogate = null
          if (codeUnit >= 0xDC00 && codeUnit <= 0xDFFF) {
            finalChunk += highSurrogate + item
            continue
          }

          finalChunk += "\uFFFD"
        }

        if (codeUnit >= 0xD800 && codeUnit <= 0xDBFF) {
          this.#pendingHighSurrogate = item
          continue
        }

        if (codeUnit >= 0xDC00 && codeUnit <= 0xDFFF) {
          finalChunk += "\uFFFD"
          continue
        }

        finalChunk += item
      }

      if (finalChunk) {
        controller.enqueue(this.#handle.encode(finalChunk))
      }
    },

    flush: controller => {
      // https://encoding.spec.whatwg.org/#encode-and-flush
      if (this.#pendingHighSurrogate !== null) {
        controller.enqueue(new Uint8Array([0xEF, 0xBF, 0xBD]))
      }
    }
  })

  get encoding() {
    return this.#handle.encoding
  }

  get readable() {
    return this.#transform.readable
  }

  get writable() {
    return this.#transform.writable
  }

  get [Symbol.toStringTag]() {
    return "TextEncoderStream"
  }
}

[PatrickJS]

looks good lets try it

[PatrickJS]

@octet-stream thanks for finding this better polyfill

[thejackshelton]

Tests pass locally and make sense on my end.

My understanding, is that this polyfill is removing the previous one, and is a more robust version that should support the 5% or so where this is not supported.

Is that the case? Anything I should look in-depth into?

Otherwise LGTM! 🙌