Puppeteer Firefox: the path to no longer being experimental

[mathiasbynens]

With https://puppeteer.github.io/ispuppeteerfirefoxready/ charts steadily going up, a question that often comes up is: when can we ship Puppeteer Firefox as non-experimental? We might not need 100% API support, if (for example) the most important 80% is covered.

This brings us to the more interesting question: what are the most crucial Puppeteer APIs for Firefox to support? Which are most commonly used? @christian-bromann @benjamingr Do you have any insights here from the SauceLabs & Testim side?

cc @mjzffr

[benjamingr]

Hmm, I think this is an interesting question but I am not sure what "most commonly used" here means.

Would a breakdown of what user actions are performed in Firefox (in WebDriver verbs) help? CDP commands in Chrome puppeteer or ChromeDriver?

Would it help if we took a large test suite (Testim's "Capabilities" that checks hundreds of open-source packages e2e) and tried running it with puppeteer-firefox?

[mjzffr]

Our team at Mozilla has gathered some data like this (e.g. https://wiki.mozilla.org/Remote/GutenbergCDPUsage). More data would be useful: protocol-level log output of a large test suite run with puppeteer against Chrome is the right way to go to get a sense of which CDP methods are used. CDP events are trickier to assess.

Similarly, we've been tracking issues in the Puppeteer repo regarding missing CDP methods in Firefox; that gives a slightly different view of what's blocking users but it's hard to tell what the scope of those bug reports is (personal projects/experiments versus more substantial work).

Anyway, having about CDP usage helps, but it's just one way to inform priorities. Other ideas are certainly welcome.

[benjamingr]

@mjzffr

Anyway, having about CDP usage helps, but it's just one way to inform priorities. Other ideas are certainly welcome.

If it helps I am happy to speak to legal (internally) and try and provide anonymized test data for large companies (Customers of Testim like Microsoft/SalesForce/Wix/Autodesk).

That could mostly give usage of what user action is performed (like clicks/keyboard/drags etc) but not CDP commands (since those run differently at Testim on different browesrs and platforms - we might be able to provide it for one though like Chrome without GPU/Linux with Xvfb but we fire different commands depending on the platform, fonts, the display and the GPU as there is a lot of variability in how commands are executed (in Chrome).

[mjzffr]

@benjamingr Sure, especially if you can find a way to provide CDP usage (even if it's for just Chrome in one type of environment, as you say).

we fire different commands depending on the platform

Is that through a custom CDP client or existing libraries like Puppeteer & Playwright... or something else?

[benjamingr]

@mjzffr

Sure,

Great, I just sent an email to the person in charge of data/legal stuff and I'll update (after the weekend).

Is that through a custom CDP client or existing libraries like Puppeteer & Playwright... or something else?

For example - Mac and Linux behave differently (more particularly input_injector_mac and input_injector_x11 behave differently and even more particularly CGPostMouseEvent and xtest fake input). This isn't special to puppeteer and we have thousands of lines of code to normalize this in webdriver as well.

Just to name a single example in Mac with a GPU if you dispatch an Input.dispatchMouseEvent with a mouseMoved over an element it will "hover" over it and :hover css will trigger. On the other hand in linux it will not and you need to dispatch a separate event first with an x/y not over the element in order to get :hover to trigger.

Automation varies by browser but also by platform, version, display and a bunch of other things - I've seen bugs in tests across environments due to different network congestion control. I hope most users don't run into this as much or as often as I do but since we support custom grids we run into the quirks of our own (open-source based off the shelf) grids and in those of most selenium grid vendors (those that provide CDP access and those who don't and use the webdriver protocol that just runs CDP commands under the hood with ChromeDriver).

[mjzffr]

As a rough idea, we've already identified that tasks related to network events, request interception, authentication, among others, probably need better support before we lift the experimental flag (see bugzilla query, but it would be good to narrow that down or learn if there other essential features.

[whimboo]

Note that with issue #6894 we will go ahead and change the recommended version of Firefox away from Nightly to release (or beta first).

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[whimboo]

@mathiasbynens not sure if we want to keep this bug open or not. As discussed as well in one of the former meetings we are not going to switch to beta/release in the foreseeable future (or at all). Focus is on WebDriver BiDi and we sadly do not have the man power to keep Puppeteer fully supported especially not for a beta or release.

[whimboo]

Maybe we should dupe to issue #6894?

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[whimboo]

Lets dupe to #6894.