Use BiDi protocol for Chrome tests #17962

calixteman · 2024-04-17T19:10:53Z

No description provided.

calixteman · 2024-04-17T19:11:29Z

/botio integrationtest

moz-tools-bot · 2024-04-17T19:11:31Z

From: Bot.io (Linux m4)

Received

Command cmd_integrationtest from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/395e307c994cbca/output.txt

moz-tools-bot · 2024-04-17T19:11:31Z

From: Bot.io (Windows)

Received

Command cmd_integrationtest from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/aead5c6b2e4df46/output.txt

timvandermeij

I'll have a detailed look at this tomorrow, but in the meantime here are my first observations.

test/test.mjs

timvandermeij · 2024-04-17T19:16:34Z

test/integration/scripting_spec.mjs

-      await Promise.all(
-        pages.map(async ([browserName, page]) => {
-          if (process.platform === "win32" && browserName === "firefox") {
-            pending("Disabled in Firefox on Windows, because of bug 1662471.");


It's really nice that we can now actually run this test everywhere now! Would it perhaps be good to keep a reference to the mentioned bug in the new comment you added, to prevent someone from changing this back later until the bug is fixed upstream?

timvandermeij · 2024-04-17T19:17:45Z

test/integration/freetext_editor_spec.mjs

@@ -2354,14 +2354,14 @@ describe("FreeText Editor", () => {
            `${getEditorSelector(1)} .overlay.enabled`
          );

-          rect = await page.$eval(getEditorSelector(0), el => {
+          rect = await page.$eval(getEditorSelector(1), el => {


Could you explain a bit how the changes in this file fix the issue? Looking at this test earlier I was a bit confused as to what this part of the test is achieving. Perhaps it could be resolved with more accompanying comments in the test to explain the intent of each "block" of logic. Because the test is quite lengthy that would also help with reading the test from top to bottom.

When taking the rect for editor 0 which now contains "A" and then clicking at rect.x+5rect.width,rect.y+5rect.height doesn't guarantee to be on the second editor.
I probably copy/paste some code without thinking about this.

moz-tools-bot · 2024-04-17T19:19:11Z

From: Bot.io (Linux m4)

Success

Full output at http://54.241.84.105:8877/395e307c994cbca/output.txt

Total script time: 7.67 mins

Integration Tests: Passed

moz-tools-bot · 2024-04-17T19:30:02Z

From: Bot.io (Windows)

Success

Full output at http://54.193.163.58:8877/aead5c6b2e4df46/output.txt

Total script time: 18.50 mins

Integration Tests: Passed

calixteman · 2024-04-17T20:03:21Z

/botio test

moz-tools-bot · 2024-04-17T20:03:23Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/9e74d8b32fd9a8a/output.txt

moz-tools-bot · 2024-04-17T20:03:24Z

From: Bot.io (Windows)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/92a94d4274292a5/output.txt

moz-tools-bot · 2024-04-17T20:08:46Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/92a94d4274292a5/output.txt

Total script time: 5.37 mins

Unit tests: FAILED
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.193.163.58:8877/92a94d4274292a5/reftest-analyzer.html#web=eq.log

moz-tools-bot · 2024-04-17T20:16:56Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/9e74d8b32fd9a8a/output.txt

Total script time: 13.53 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.241.84.105:8877/9e74d8b32fd9a8a/reftest-analyzer.html#web=eq.log

calixteman · 2024-04-18T08:00:19Z

@OrKoN the ref tests failed on the linux bot because of an OOM crash. The bot has a 64 bits OS with 16Gb of RAM and afaik we never had this kind of issues with CDP in the last years.
And on the windows bot, it failed in the middle of the unit test with an exception (TargetCloseError: Protocol error (Runtime.callFunctionOn): Target closed).

calixteman · 2024-04-18T11:21:01Z

/botio test

moz-tools-bot · 2024-04-18T11:21:03Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/17e8528589e75e3/output.txt

moz-tools-bot · 2024-04-18T11:21:03Z

From: Bot.io (Windows)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.193.163.58:8877/81255050207bcff/output.txt

moz-tools-bot · 2024-04-18T11:31:00Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/17e8528589e75e3/output.txt

Total script time: 9.94 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.241.84.105:8877/17e8528589e75e3/reftest-analyzer.html#web=eq.log

OrKoN · 2024-04-18T11:50:33Z

@calixteman do all tests use Puppeteer or only integration?

OrKoN · 2024-04-18T11:51:01Z

I think I am not able to view the logs but I will give it a try later

moz-tools-bot · 2024-04-18T12:00:59Z

From: Bot.io (Windows)

Failed

Full output at http://54.193.163.58:8877/81255050207bcff/output.txt

Total script time: 39.92 mins

Unit tests: FAILED
Integration Tests: FAILED
Regression tests: FAILED

  different ref/snapshot: 5

Image differences available at: http://54.193.163.58:8877/81255050207bcff/reftest-analyzer.html#web=eq.log

calixteman · 2024-04-18T12:03:58Z

@calixteman do all tests use Puppeteer or only integration?

The unit/ref/integration tests are using Puppeteer.

calixteman · 2024-04-18T12:05:14Z

I think I am not able to view the logs but I will give it a try later

You should be able to just look at http://54.193.163.58:8877/81255050207bcff/output.txt

timvandermeij · 2024-04-18T12:44:45Z

@OrKoN The logs of the runs that crashed due to OOM can be found at http://54.241.84.105:8877/9e74d8b32fd9a8a/output.txt for the first try and at http://54.241.84.105:8877/17e8528589e75e3/output.txt for the second try. Both crashed during the reference tests. For completeness I'll also include the relevant bit here (the TEST-PASS line is the last bit of the PDF.js code):

TEST-PASS | eq test issue13751 | in firefox

<--- Last few GCs --->

[4032:0x6eba470]   422579 ms: Scavenge (reduce) 4087.8 (4127.6) -> 4087.9 (4128.6) MB, 2.1 / 0.0 ms  (average mu = 0.145, current mu = 0.047) allocation failure; 
[4032:0x6eba470]   422585 ms: Scavenge (reduce) 4088.8 (4128.6) -> 4088.7 (4129.3) MB, 3.1 / 0.0 ms  (average mu = 0.145, current mu = 0.047) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb7a940 node::Abort() [node]
 2: 0xa8e823  [node]
 3: 0xd5c940 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xd5cce7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xf3a3e5  [node]
 6: 0xf3b2e8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xf4b7f3  [node]
 8: 0xf4c668 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf26fce v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xf28397 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
11: 0xf088e0 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
12: 0xf00354 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node]
13: 0xf02608 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node]
14: 0x1042959 v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [node]
15: 0x1044676 v8::internal::JsonParser<unsigned short>::ParseJsonValue() [node]
16: 0x104515f v8::internal::JsonParser<unsigned short>::ParseJson() [node]
17: 0xde2523 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [node]
18: 0x16fb7b9  [node]
[11:30:57] 'runBotTest' errored after 8.62 min
[11:30:57] Error: Running browser tests failed.
    at ChildProcess.<anonymous> (file:///home/ubuntu/pdfjs/botio-files-pdfjs/private/17e8528589e75e3/gulpfile.mjs:731:15)
    at ChildProcess.emit (node:events:513:28)
    at ChildProcess.emit (node:domain:552:15)
    at maybeClose (node:internal/child_process:1091:16)
    at ChildProcess._handle.onexit (node:internal/child_process:302:5)
    at Process.callbackTrampoline (node:internal/async_hooks:130:17)

calixteman · 2024-04-18T13:11:36Z

/botio-linux test

moz-tools-bot · 2024-04-18T13:11:38Z

From: Bot.io (Linux m4)

Received

Command cmd_test from @calixteman received. Current queue size: 0

Live output at: http://54.241.84.105:8877/4340ac09990c704/output.txt

moz-tools-bot · 2024-04-18T13:21:01Z

From: Bot.io (Linux m4)

Failed

Full output at http://54.241.84.105:8877/4340ac09990c704/output.txt

Total script time: 9.37 mins

Unit tests: Passed
Integration Tests: FAILED
Regression tests: FAILED

Image differences available at: http://54.241.84.105:8877/4340ac09990c704/reftest-analyzer.html#web=eq.log

timvandermeij · 2024-04-19T17:08:32Z

Would it perhaps be an idea to pull the test improvements out into a separate PR so we can already land them while we're figuring out what to do with the Chrome/WebDriver BiDi OOM bot issue?

It would make this PR smaller and we can then already benefit from the improved code, and run the previously disabled test on Windows. Together with the waitForTimeout removals I'm hoping that all these stability improvements will fix most of the flakiness in the integration tests and make debugging any remaining issues easier.

The Puppeteer update should in particular be helpful for us because it contains improved WebDriver BiDi compatibility, a newer Chrome version (both might help for mozilla#17962) and an official deprecation of CDP for Firefox. Note that the latter doesn't require changes on our end because we already use WebDriver BiDi unconditionally for Firefox since commit 4db0174. The full release notes can be found at https://github.com/puppeteer/puppeteer/releases/tag/puppeteer-core-v22.8.0.

calixteman requested a review from timvandermeij April 17, 2024 19:10

timvandermeij added the test label Apr 17, 2024

timvandermeij reviewed Apr 17, 2024

View reviewed changes

calixteman force-pushed the chrome_bidi branch from c6c48e6 to c583cb9 Compare April 17, 2024 20:03

calixteman mentioned this pull request Apr 18, 2024

Use WebDriver BiDi for Chrome #17961

Open

Use BiDi protocol for Chrome tests

743d6a3

calixteman force-pushed the chrome_bidi branch from c583cb9 to 743d6a3 Compare April 18, 2024 13:11

This was referenced Apr 30, 2024

Fix flickering on text selection #17923

Merged

Update dependencies and translations to the most recent versions #18053

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use BiDi protocol for Chrome tests #17962

Use BiDi protocol for Chrome tests #17962

calixteman commented Apr 17, 2024

calixteman commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

timvandermeij left a comment

timvandermeij Apr 17, 2024 •

edited

timvandermeij Apr 17, 2024 •

edited

calixteman Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

calixteman commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

calixteman commented Apr 18, 2024

calixteman commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

OrKoN commented Apr 18, 2024

OrKoN commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

calixteman commented Apr 18, 2024

calixteman commented Apr 18, 2024

timvandermeij commented Apr 18, 2024 •

edited

calixteman commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

timvandermeij commented Apr 19, 2024

Use BiDi protocol for Chrome tests #17962

Are you sure you want to change the base?

Use BiDi protocol for Chrome tests #17962

Conversation

calixteman commented Apr 17, 2024

calixteman commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Windows)

Received

timvandermeij left a comment

Choose a reason for hiding this comment

timvandermeij Apr 17, 2024 • edited

Choose a reason for hiding this comment

timvandermeij Apr 17, 2024 • edited

Choose a reason for hiding this comment

calixteman Apr 17, 2024

Choose a reason for hiding this comment

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Linux m4)

Success

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Windows)

Success

calixteman commented Apr 17, 2024

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Windows)

Received

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Windows)

Failed

moz-tools-bot commented Apr 17, 2024

From: Bot.io (Linux m4)

Failed

calixteman commented Apr 18, 2024

calixteman commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Windows)

Received

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Linux m4)

Failed

OrKoN commented Apr 18, 2024

OrKoN commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Windows)

Failed

calixteman commented Apr 18, 2024

calixteman commented Apr 18, 2024

timvandermeij commented Apr 18, 2024 • edited

calixteman commented Apr 18, 2024

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Linux m4)

Received

moz-tools-bot commented Apr 18, 2024

From: Bot.io (Linux m4)

Failed

timvandermeij commented Apr 19, 2024

timvandermeij Apr 17, 2024 •

edited

timvandermeij Apr 17, 2024 •

edited

timvandermeij commented Apr 18, 2024 •

edited