[Bug]: Final status code changes when request intercept is TRUE

[kazanemed]

Minimal, reproducible example

            await page.setRequestInterception(true);
            await page.on(PageEmittedEvents.Request, (req) => {
            req.continue();
            });

            const response = await Promise.race([
              page?.goto("https://bit.ly/1gqbFUR", { waitUntil: ['networkidle0'] }),
              page.waitForTimeout(TIMEOUT_MS_SCREENSHOT).then(() => {
                res.status(500).json({ message: TIMEOUT_ERROR_MESSAGE });
              })
            ]);

            console.log(response.status())

Error string

no error

Bug behavior

• Flaky
• PDF

Background

When i set request interceptor ( even without logic inside ) and i do console log for the destination url status code of websites that has one of the following redirects statuses of 301, 302, 307 or 308 in the redirect chain, it gives me that the final status is either 301, 302, 307, 308 instead of the real destination page status code 200 as shown in my browser network tab for this website: https://bit.ly/1gqbFUR . But when i do remove Request Interceptor, i get the 200 status code.

Expectation

For this website as an example that has a 301 redirect : "https://bit.ly/1gqbFUR
", result of final url status code should be 200

Reality

I'm getting 301

Puppeteer configuration file (if used)

No response

Puppeteer version

21.10.0

Node version

18.17.0

Package manager

npm

Package manager version

9.6.7

Operating system

macOS

[github-actions]

This issue has an outdated Puppeteer version: 21.10.0. Please verify your issue on the latest 22.6.2 version. Then update the form accordingly.

---

Analyzer run

[kazanemed]

I upgraded to 22.6.2 and still same issue.

[kazanemed]

@OrKoN @Lightning00Blade any news about this? It's an urgent bug that need to be fixed in our project which is using puppeter. Thank you in advance.

[OrKoN]

@kazanemed no updates here, we are happy to review a pull request if you find what is going wrong here. It is also possible that the issue is upstream in Chromium.

[Lightning00Blade]

I have tracked down the issue and it's that there is a bug in Chromium where there is no interception for an update from insecure to secure (header: "Upgrade-Insecure-Requests":"1"), and Puppeteer expects them to happen, thus we have one less we don't consume one of the events (Network.responseReceivedExtraInfo). I am not sure we can action on it from our side.

For safekeeping Log Gist

Test in Puppeteer

  it('redirect interception issue', async () => {
    const {page} = await getTestState();
    await page.setRequestInterception(true);
    page.on('request', req => {
      void req.continue();
    });
    const responses: HTTPResponse[] = [];
    page.on('response', res => {
      if (res.request().isNavigationRequest()) {
        responses.push(res);
      }
    });
    const response = await page.goto('https://bit.ly/1gqbFUR', {
      waitUntil: ['networkidle0'],
    });

    console.log(responses.at(-1)?.status());
    expect(response?.status()).toBe(200);
  });

[OrKoN]

@Lightning00Blade the response received event has the 200 status but we do not detect that. It looks like we should be able to extra the right status even if some extra info events are missing?

[Lightning00Blade]

We have tracked down the issue to HttpsUpgrades.
I will file an issue upstream and update this comment with it.
If you disable it, you should stop seeing this.

const browser = puppeteer.launch({
  args: ["--disable-features=HttpsUpgrades"]
})

[kazanemed]

We have tracked down the issue to HttpsUpgrades. I will file an issue upstream and update this comment with it. If you disable it, you should stop seeing this.

const browser = puppeteer.launch({
  args: ["--disable-features=HttpsUpgrades"]
})

This works perfectly! Thank you @Lightning00Blade .