Better Support for Shadow DOM

[aslushnikov]

Since Copy JS Path has landed in Chrome DevTools, we now have a way to address nodes inside shadow DOM.

Details: #858 (comment)

This, however, doesn't simplify life much when it comes to sugar methods, such as page.click, page.select and others.

It'd be nice to have a way to address nodes in shadow DOM in these methods.

Suggestions:

1. Teach page.click and others to accept an array of selectors. (source).

await page.select(['#container', 'select#foo'], 'value');

2. Teach page.click and others to treat jsPaths as selectors.

await page.select("document.querySelector('body > toolbar-component > toolbar-section.left')", 'value');

cc @JoelEinbinder

[felixfbecker]

Another thing to consider that in an application where every (web)component is a shadow root, this would basically be equivalent to an exact step by step DOM path instead of a "selector". This means you would have to update all your e2e tests every time you change anything about the DOM structure, which would be extremely annoying (no matter if it supports JS path or arrays).

Maybe another better option would be to have an option { penetrateShadowRoot: true } to the functions that would recursively dig into shadow roots.

[Jamesernator]

Could the >>> selector be reintroduced but only allowed via the devtools protocol?

[felixfbecker]

Btw, how do XPath queries behave with regards to shadow roots?

[Jamesernator]

Also with the new introduction of ::part it would make sense if page.click() and friends accepted an option to select a specific ::part.

[JoelEinbinder]

await page.click(() => window.mySuperCoolElement)

[elf-pavlik]

Maybe page.waitForSelector(selector[, options]) could also take JS Path?
How one supposed currently to wait for selectors in shadow dom?

[Georgegriff]

I feel like most of solutions to shadow Dom in puppeteer force the tests/tester to have to know far too much about the structure of the application.

Having page.click work with jsPath would make a library I wrote a bit easier to work with. (Below)

The purpose of the library is to allow you to just write a normal querySelector and it will automatically find the selector whether it's in a shadow root or deeply nested shadow roots.

For now I use evaluateHandle and return the ElementHandle which you can do clicks, types, select etc on.

https://github.com/Georgegriff/query-selector-shadow-dom/blob/master/examples/puppeteer/clicking-elements.js

I've thought about wrapping the above snippet into a library that lets you pass in a puppeteer page object and a querySelector.

[thernstig]

Having experimented with this, and added my custom waitForElementShadowDOM I do believe the best approach to add this natively into page. functions is to go with the suggestion that @felixfbecker has. I'd even say that a generic/page/browser-wide setting to add { penetrateShadowRoot: true } would make the most sense. Reason being that users that e.g. use Polymer (or rather lit-html) to build their whole page will have a lot of shadowRoots.

Having to specify the exact DOM tree with an array, or with other means such as what JSPath requires, is not ideal. Imagine you move around a generic component called <some-generic-component> to another part of the DOM tree. It means one would have to update all arrays/JSPaths that points to it over a bunch of test suites (e.g. if you just Jest+Puppeteer to do e2e testing).

Moreover, if you want to test it on various viewport sizes, the actual DOM can look different which means tests will not work anyway.

@aslushnikov Considering Web Components is becoming more used, is there a plan to natively implement penetrating shadow DOMs into Pupeteer, or is it a low priority task?

[thernstig]

@aslushnikov Is there any planned worked for this from the Puppeteer team? Considering about %8 of page loads use custom elements (and most likely shadow dom) it feels like a prominent framework such as Puppeteer could use a more native approach :)

[doriansmiley]

If there is anyone out there who was trying to do SSR with web components I finally figured out how to support this in puppeteer. You need to use Declarative Shadow DOM. In your template add shadowroot ie <template shadowroot="open">. Then when you launch Puppeteer:

const args = puppeteer.defaultArgs();
    // IMPORTANT: you can't render shadow DOM without this flag
    // getInnerHTML will be undefined without it
    args.push('--enable-experimental-web-platform-features');

Then add a waitForFunction:

await page.waitForFunction(() => !!document.querySelector('myRoot')?.shadowRoot, {
            polling: 'mutation',
        });

Then to serialize you need:

const html = await page.$eval('html', (element) => {
            return element.getInnerHTML({includeShadowRoots: true});
        });

The getInnerHTML is part of experimental APIs. If you want to test in your browser: chrome://flags/#enable-experimental-web-platform-features. Be sure to read the linked article for a complete explanation.

[JoCa96]

Since #6509 and 5.4.0 there is now a pierce handler.
You can use it like this:

it('should find all elements in shadow', async () => {
  const { page } = getTestState();
  const divs = await page.$$('pierce/.foo');
  const text = await Promise.all(
    divs.map((div) =>
      div.evaluate((element: Element) => element.textContent)
    )
  );
  expect(text.join(' ')).toBe('Hello World');
});

Or you can have a look at the test example in the original PR.
But I didn't find anything about this in the docs, so it would be good if they would be updated.