Change recommended version of Firefox from Nightly to Beta/Release

[whimboo]

As of right now Puppeteer downloads Firefox Nightly by default. This was pretty helpful for us while getting started to implement CDP commands in Firefox, and get new features out as soon as possible.

But that also have some side-effects as what we have seen in the past months. Firefox Nightly is not meant to be that stable, and also has various experiments enabled that can badly affect the user experience. As such I would propose that we change Puppeteer to no longer download Firefox Nightly by default but a release version (or beta?) of Firefox. Note that with the upcoming Firefox 86 release next week the remote agent will be available by default.

Due to an oversight on my side we missed to enable our tests on those channels. As such maybe we could go with Firefox 87 Beta first, or wait a little bit longer so we can get confidence that everything works as expected? @mathiasbynens what do you think?

To prepare the change, which specific areas would have to be changed in Puppeteer to make that move?

CC'ing @jackfranklin, @jschfflr, @juliandescottes, @jgraham, @janodvarko

[mathiasbynens]

IMHO anything would be better than the status quo, which uses "whatever the latest Nightly is". Preparing a change now, to go with Firefox 87 beta first, sounds good to me.

In terms of which Puppeteer changes would be needed, this would boil down to updating
https://github.com/puppeteer/puppeteer/blob/b046fd79bd97e8623c40204193bff028096e4474/src/node/BrowserFetcher.ts. Happy to review a patch!

[whimboo]

Thanks. Meanwhile I also got our tests enabled for beta and these are all fine. So I will try with 87 beta first next week.

One problem we might hit is that we usually do not offer zip archives of releases on Windows. I would have to check where I can get these from.

[whimboo]

Internally I will track the work via https://bugzilla.mozilla.org/show_bug.cgi?id=1694127.

@mathiasbynens would it be also ok to use an exe or msi installer? We officially don't have zip archives of releases, but only for candidate build, which I wouldn't like to fallback to. If that's ok hopefully Node.js has some package to do a silent installation.

[mathiasbynens]

My gut reaction is that it seems a little intrusive to download and run .exes upon npm install puppeteer. The expectation is that node some-puppeteer-script.js can launch browser binaries, but not that npm install puppeteer does it — and I feel uncomfortable changing that.

Would it be possible for Mozilla to provide ZIP archives of releases? That seems like the smallest possible solution from the Puppeteer side.

[whimboo]

No, there won't be zip archives for releases. And that decision has been made a while ago, and our case with Puppeteer will not change it.

So actually both the .exe and .msi installers can run in silent mode, and could be run to just extract their content. That means no user interaction is necessary. Here is how we do it in our Python harnesses that also accept .exe and .msi:

https://searchfox.org/mozilla-central/rev/c8ce16e4299a3afd560320d8d094556f2b5504cd/testing/mozbase/mozinstall/mozinstall/mozinstall.py#345-347

Maybe that is a viable alternative given that it doesn't run any installer?

[whimboo]

@mathiasbynens and @jschfflr maybe you can give me feedback to my last comment? Thanks.

[jschfflr]

To me, it sounds like the best available option then. But I'm not sure how @mathiasbynens will feel about that.

[mathiasbynens]

No, there won't be zip archives for releases. And that decision has been made a while ago, and our case with Puppeteer will not change it.

What's the rationale behind this? Is there any publicly-accessible background? Would love to understand this better.

I stand by my earlier comment, and to make a stronger claim: from a security perspective, it’s not an acceptable option for Puppeteer to run a remotely downloaded .exe on npm install puppeteer. It doesn’t matter whether the executable has a GUI or not.

[whimboo]

If you want to read details you can have a look at https://bugzilla.mozilla.org/show_bug.cgi?id=308048#c6. That's the only public resource that I can link to. Ever since we ship releases as .exe and .msi.

As it sounds like it's a showstopper for you, the only other option that we could offer is to get the final candidate build before it becomes a release. While builds are stored at locations like https://archive.mozilla.org/pub/firefox/candidates/91.0-candidates/build2/win64/en-US/ we should use CDN for the download.

[mathiasbynens]

As it sounds like it's a showstopper for you, the only other option that we could offer is to get the final candidate build before it becomes a release. While builds are stored at locations like https://archive.mozilla.org/pub/firefox/candidates/91.0-candidates/build2/win64/en-US/ we should use CDN for the download.

This sounds great! ZIPs seem to be available at that location. I don’t know what the CDN URLs would look like but I guess I’ll find out once y’all send a patch 👍

[jschfflr]

@whimboo Any updates on this?

[whimboo]

Sorry, I had PTO for most of August, and since I'm back we are trying to find a good way in getting a download of an archive accomplished. I should have an update later this week.

[whimboo]

So with the requirement to only download archives from Firefox releases the only option we see to get to the right build URLs is the following (also see bug 1694127):

1. Find the LATEST_FIREFOX_VERSION: https://product-details.mozilla.org/1.0/firefox_versions.json
2. Find the last build_number: https://shipitapi-public.services.mozilla.com/releases?product=firefox&status=shipped&version=%LATEST_FIREFOX_VERSION%
3. Download the build from https://archive.mozilla.org/pub/firefox/candidates/%LATEST_FIREFOX_VERSION%-candidates/build%build_number%/%platform%/en-US/ whereby file extensions vary per platform between firefox-%LATEST_FIREFOX_VERSION%.(tar.bz2|zip) or Firefox%20%LATEST_FIREFOX_VERSION%.dmg

Is that acceptable for the Puppeteer project? Both API's as used under 1. and 2. are public and stable.

[mathiasbynens]

@whimboo Sounds good to me. Would you want to hardcode the build numbers (so that every Puppeteer release defaults to a specific Firefox version, like we do it for Chromium binaries) or always get the latest version (like we’ve been doing until now for the experimental Firefox support)? If we’d prefer hardcoding, we could simply skip your suggested step 1.

[jschfflr]

I think fixing the version number could be helpful for situations like this one: #7568

Sorry, I had PTO for most of August, and since I'm back we are trying to find a good way in getting a download of an archive accomplished. I should have an update later this week.

Hope you had a great time off!

[whimboo]

Hard-coding the Firefox version number might be good, even with being more flexible (backward compatible) with our releases so far. But I would allow to fallback to a more recent security release if available. Is that something you also handle for Chrome / Chromium?

Note that #7568 was for a Nightly build and should never appear for a release. But I would be more afraid of breakage due to backward incompatible CDP API changes. In those cases a clear signoff on our side for the new release might be good.

[whimboo]

So one more question... Chrome and Firefox releases don't happen at the same time. What if a backward incompatible change gets introduced in Firefox after a Puppeteer release? I assume it would need a follow-up Puppeteer minor release?

[jschfflr]

I wouldn't mind cutting a new Puppeteer release in these cases. At the moment, there is no clear mapping between Chromium revisions and Puppeteer versions.

[whimboo]

The work on this issue will be delayed for a bit. Reason is that soon our Fission (site isolation) support will be on by default, and soon after cannot be disabled. This would break Firefox users, and as such I want to see that fixed first. Once done we could move the binaries to release ones.

[whimboo]

Also as known the CI jobs on Windows fail a lot with Firefox right now. I did and continue to do investigations in how to improve that. Work for that is done on #7668. As such there might be more blockers that we have to take into account that would not allow us to make this change as requested on this issue anytime soon. But lets see how quick we can get these fixes into a release build of Firefox.

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[whimboo]

Not sure if we want to keep this issue open or not. As discussed as well in one of the former meetings we are not going to switch to beta/release in the foreseeable future (or at all). Focus is on WebDriver BiDi and we sadly do not have the man power to keep Puppeteer fully supported especially not for a beta or release.

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.