Puppeteer E2E test: Fix text-rendering differences

[LeviPesin]

Related:

• Puppeteer E2E test: Regenerate high-differenсe screenshots and decrease threshold #25403 (comment)
• Achieve consistent font rendering between different platforms. puppeteer/puppeteer#661
• Inconsistent text rendering in headless mode puppeteer/puppeteer#2410

Description

Fix differences in text rendering in Puppeteer between different platforms and also regenerate some screenshots (using Chromium 109.0.5414.120 (revision 1070054, https://storage.googleapis.com/chromium-browser-snapshots/Win_x64/1070054/chrome-win.zip), stable channel on win64).

[LeviPesin]

It doesn't really seem this change works...

[epreston]

@LeviPesin gday, e2e was failing all tests on the mac after 3 mins so I took a peek. Code is pushing an --enable-features=Vulkan flag. That doesn't look right. What's it required for ?

Also added a --enable-logging=stderr --v=1 to the build flags to get more detailed console messages during runs so I can pass along detailed reasons for failures.

Where do we tune the memory allocated for the chromium browser session ?

[epreston]

Your constant for mac_arm , in PLATFORMS is missing an R in the value.

should be:

const PLATFORMS = {
	linux: 'linux',
	mac: 'mac',
	mac_arm: 'mac_arm64',
	win32: 'win',
	win64: 'win64'
};

See: #25395

[epreston]

To get the native mac browser and avoid slow rosetta emulation (buggy because its never tested on arm because they have arm builds) looks like you need to opt into it:

const browserFetcher = new BrowserFetcher( { path: 'test/e2e/chromium', useMacOSARMBinary: true } );

After fixing the typo in the PLATFORMS constant and opting into arm builds (old BrowserFetcher lib ?) :

console.log( browserFetcher.platform() );

This will return mac_arm instead of mac when it's on ARM. In turn, that downloaded the chromium for mac_arm and started running tests.

[epreston]

It's finally able to complete a full run. The messages at the end might not be what you want to do.

webgl_multiple_elements_text error was:

webgl_postprocessing_procedural error was:

I am doing another pull and running again. I see there's been some more updates.

[LeviPesin]

Your constant for mac_arm , in PLATFORMS is missing an R in the value.

To get the native mac browser and avoid slow rosetta emulation (buggy because its never tested on arm because they have arm builds) looks like you need to opt into it:
const browserFetcher = new BrowserFetcher( { path: 'test/e2e/chromium', useMacOSARMBinary: true } );

Thank you! Will fix those errors.

The problem why some examples fail in this PR (which I tried to fix but to no avail) is text rendering -- I tried using some flags and/or setting a custom font so that it overwrites the default system font but that doesn't seem to work. I will investigate more.

[epreston]

Ran again with all the latest updates.

2 Mods applied, one ommitted.

• PLATFORMS
• useMacOSARMBinary: true
• remove --enable-features=Vulkan

Next test is all 3. Vulkan won't have much supports on the Macs.

Updating this comment with the results. It's at least completing but now we have 5 errors instead of 2.

Results Summary:

Error Captures:

[epreston]

Update: Ignore this

Also a new issue popped up.

• When it is retrying the test, it doesn't say if it ran the test again or its status

Update: end of queue puts it further down as you said, ignore this one.

[LeviPesin]

When it is retrying the test, it doesn't say if it ran the test again or its status

This is strange, it should... It can be significantly below that line -- because when making a new attempt it's added to the end of the queue.

[epreston]

3 Mods applied:

• PLATFORMS
• useMacOSARMBinary: true
• remove --enable-features=Vulkan

First Test Run

Second Test Run

Third Test Run

[epreston]

Looks like the vulkan is harmless.

[epreston]

When it is retrying the test, it doesn't say if it ran the test again or its status

This is strange, it should... It can be significantly below that line -- because when making a new attempt it's added to the end of the queue.

Ok. Checking logs again to confirm. You're right... it was further down.

[LeviPesin]

The Vulkan flag was added in #25304 by @mrdoob. I'm not really sure about what the flags from that PR do (I think swiftshader is enabled by default for headless Chromium) but at least they fix the "no adapters error" for WebGPU on Linux.

[epreston]

Forcing swiftshader is forcing software emulation. For automated tests running in the cloud with no GPU, might be a good thing. Slow for local testing but, for one frame thats a matter of the milliseconds.

I have the same tests failing all runs. Hows this match up with the image updates and tolerances ?

[epreston]

I'm going to run the suite full native, raw, no swiftshader and post the results below so you have that data point.

Update:

Same 5. New message on webgl_modifier_simplifier

[LeviPesin]

I have the same tests failing all runs. Hows this match up with the image updates and tolerances ?

The examples that are failing are exactly the examples that use many text -- and the difference is exactly in text rendering. I will think more about how to solve this...

[epreston]

Updated the raw test data. Is it a font thing ? I remember needing to supply fonts in a few formats to get the same look across browsers.

[epreston]

The test rig is filtering web requests for the three library so it can return a version where random replaced with a fixed value to get consistent results.

These examples might import from other places. Is it possible some of these could need filtering now or in the future ?

	{
		"imports": {
			"three": "../build/three.module.js",
			"three/addons/": "./jsm/"
		}
	}

[epreston]

Another one:

Looks like the filtering is only doing http: requests. What if someone where to swap that to https in an example ? Might be required for certain types of code now or in the future.

[epreston]

The same examples fail each time. I'm going to run each example directly on the Mac and post the results.

Maybe there will be a font not loaded or not compatible, some sort of message.

[LeviPesin]

There was a discussion about Math.random() filtering in #24109. I'm not very convinced it is needed (it was said there that otherwise many screenshots could break when another Math.random() is inserted before a one)...

By the way, we can theoretically just set the JS random value when starting the browser (there is such a flag) and then remove the filtering and Math._random() code from deterministic-injection.js.

[epreston]

Half way through the Mac tests. My back was hurting so had to break from using the laptop. I'll send through the infos shortly.