Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[馃悰 Bug]: driver.execute_script() does not work with chromedriver/msedgedriver v96 - return dict and not WebDriver object #10019

Closed
fenchu opened this issue Nov 9, 2021 · 11 comments

Comments

@fenchu
Copy link

fenchu commented Nov 9, 2021

What happened?

After updating chrome and/or msedge to v96, driver.execute_script() no longer works it returns a dict(hash) instead of webDriver element:

I need to execute a shadowRoot pseudo-element in order to get the properties inside the element.

I think there are changes in the browser driver that prevent this from working,
I think this can be reproduceable with any pseudo-elements that needs javascript executing.

I've tried all combinations of:

options.w3c = True|False
options.add_experimental_option('w3c', True|False)

Current workaround is not updating beyond v95 of chrome/msedge

How can we reproduce the issue?

Example code python link to one of our customer test logins, trying to click a button behins a jsdom.

#!/usr/bin/env python3
import sys
import selenium
import selenium.webdriver
from selenium.webdriver.common.by import By

url = "https://www.trk.qa-04.buypass.no/key-registration"

# options = selenium.webdriver.EdgeOptions()
# options.binary_location = "C:\\Program Files (x86)\\Microsoft\\Edge Dev\\Application\\msedge.exe"
# #options.w3c = False
# #options.add_experimental_option('w3c', True)
# options.use_chromium = True
# wd = selenium.webdriver.Edge(options=options, service_args=['--verbose'])


# default chrome
options = selenium.webdriver.ChromeOptions()
options.binary_location = "C:\\Program Files (x86)\\Google\\Chrome Beta\\Application\\chrome.exe"
#options.w3c = False
#options.add_experimental_option('w3c', True)
wd = selenium.webdriver.Chrome(options=options)

wd.set_window_size(800,800)
print(f"webdriver:{wd.name}, capabilities:{wd.capabilities}")
print(f"type:{type(wd)}")
print(f"python version:{sys.version}")
print(f"selenium version:{selenium.__version__}")

wd.get(url)
el = wd.find_element(By.XPATH, "//bp-button[@text='Buypass ID']")
shadow_root = wd.execute_script(f"return arguments[0].shadowRoot", el)
print(f"shadow_root:{shadow_root}")
shadow_root.find_element(By.CSS_SELECTOR, "button").click()

Relevant log output

chrome v95 works as all previous versions:

python .\bin\chrome-selenium-execute-bug.py

DevTools listening on ws://127.0.0.1:54262/devtools/browser/7904da2f-0317-49fb-a07d-2e7706c20324
[15352:11568:1109/133205.386:ERROR:display_layout.cc(559)] PlacementList must be sorted by first 8 bits of display_id 
[15352:11568:1109/133205.700:ERROR:display_layout.cc(559)] PlacementList must be sorted by first 8 bits of display_id 
webdriver:chrome, capabilities:{'acceptInsecureCerts': False, 'browserName': 'chrome', 'browserVersion': '95.0.4638.69', 'chrome': {'chromedriverVersion': '95.0.4638.17 (a9d0719444d4b035e284ed1fce73bf6ccd789df2-refs/branch-heads/4638@{#178})', 'userDataDir': 'C:\\Users\\mobj\\AppData\\Local\\Temp\\scoped_dir14728_516592648'}, 'goog:chromeOptions': {'debuggerAddress': 'localhost:54262'}, 'networkConnectionEnabled': False, 'pageLoadStrategy': 'normal', 'platformName': 'windows', 'proxy': {}, 'setWindowRect': True, 'strictFileInteractability': False, 'timeouts': {'implicit': 0, 'pageLoad': 300000, 
'script': 30000}, 'unhandledPromptBehavior': 'dismiss and notify', 'webauthn:extension:credBlob': True, 'webauthn:extension:largeBlob': True, 'webauthn:virtualAuthenticators': True}
type:<class 'selenium.webdriver.chrome.webdriver.WebDriver'>
python version:3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]
selenium version:4.0.0
shadow_root:<selenium.webdriver.remote.webelement.WebElement (session="42ee4e062c26b95006696a89773fd37e", element="c1b47141-61c3-40bc-88e8-0fa2ea2cde5b")>

chrome v96 does not work:

python .\bin\chrome-selenium-execute-bug.py
DevTools listening on ws://127.0.0.1:55116/devtools/browser/67905cf2-00b9-4455-9744-6ccd48f654a0
[12352:21508:1109/140012.902:ERROR:display_layout.cc(559)] PlacementList must be sorted by first 8 bits of display_id 
[12352:21508:1109/140013.144:ERROR:display_layout.cc(559)] PlacementList must be sorted by first 8 bits of display_id 
webdriver:chrome, capabilities:{'acceptInsecureCerts': False, 'browserName': 'chrome', 'browserVersion': '96.0.4664.35', 'chrome': {'chromedriverVersion': '96.0.4664.35 (8df3babf8da8faa8aacb78f4848ef66a93b4944b-refs/branch-heads/4664@{#680})', 'userDataDir': 'C:\\Users\\mobj\\AppData\\Local\\Temp\\scoped_dir15676_76951379'}, 'goog:chromeOptions': {'debuggerAddress': 'localhost:55116'}, 'networkConnectionEnabled': False, 'pageLoadStrategy': 'normal', 'platformName': 'windows', 'proxy': {}, 'setWindowRect': True, 'strictFileInteractability': False, 'timeouts': {'implicit': 0, 'pageLoad': 300000, 'script': 30000}, 'unhandledPromptBehavior': 'dismiss and notify', 'webauthn:extension:credBlob': True, 'webauthn:extension:largeBlob': True, 'webauthn:virtualAuthenticators': True}
type:<class 'selenium.webdriver.chrome.webdriver.WebDriver'>
python version:3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]
selenium version:4.0.0
shadow_root:{'shadow-6066-11e4-a52e-4f735466cecf': '07cda0b2-3936-4421-91fa-c0a69d96aa93'}
Traceback (most recent call last):
  File "C:\dist\work\trk-fullstack-test\bin\chrome-selenium-execute-bug.py", line 31, in <module>
    shadow_root.find_element(By.CSS_SELECTOR, "button").click()
AttributeError: 'dict' object has no attribute 'find_element'

msedgev97 does not work:

python .\bin\chrome-selenium-execute-bug.py
[264:11416:1109/134247.963:ERROR:clean_exit_beacon.cc(241)] beacon_file_path_: C:\Users\mobj\AppData\Local\Temp\scoped_dir3512_1790376771\Variations
[264:11416:1109/134247.966:ERROR:clean_exit_beacon.cc(141)] got_beacon_file_contents: 0

DevTools listening on ws://127.0.0.1:54551/devtools/browser/c5b4e18b-57bc-4eb0-be57-5a1dbe9b5438
 [264:11416:1109/134249.808:ERROR:fallback_task_provider.cc(118)] Every renderer should have at least one task provided by a primary task provider. If a fallback task is shown, it is a bug. Please file a new bug and tag it as a dependency of crbug.com/739782.
webdriver:msedge, capabilities:{'acceptInsecureCerts': False, 'browserName': 'msedge', 'browserVersion': '97.0.1060.2', 'ms:edgeOptions': {'debuggerAddress': 'localhost:54551'}, 'msedge': {'msedgedriverVersion': '97.0.1060.2 (0365cba3e5dc1a8b30b892df0cf076eee17919ce)', 'userDataDir': 'C:\\Users\\mobj\\AppData\\Local\\Temp\\scoped_dir3512_1790376771'}, 'networkConnectionEnabled': False, 'pageLoadStrategy': 'normal', 'platformName': 'windows', 'proxy': {}, 'setWindowRect': True, 'strictFileInteractability': False, 'timeouts': {'implicit': 0, 'pageLoad': 300000, 'script': 30000}, 'unhandledPromptBehavior': 'dismiss and notify', 'webauthn:extension:credBlob': True, 'webauthn:extension:largeBlob': True, 'webauthn:virtualAuthenticators': True}
type:<class 'selenium.webdriver.edge.webdriver.WebDriver'>
python version:3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]
selenium version:4.0.0
[264:11416:1109/134251.810:ERROR:fallback_task_provider.cc(118)] Every renderer should have at least one task provided by a primary task provider. If a fallback task is shown, it is a bug. Please file a new bug and tag it as a dependency of crbug.com/739782.
shadow_root:{'shadow-6066-11e4-a52e-4f735466cecf': 'a6a68841-eb92-420e-9918-347bf9a70814'}
Traceback (most recent call last):
  File "C:\dist\work\trk-fullstack-test\bin\chrome-selenium-execute-bug.py", line 30, in <module>
    shadow_root.find_element(By.CSS_SELECTOR, "button").click()
AttributeError: 'dict' object has no attribute 'find_element'

Operating System

windows10, ubuntu20.04LTS

Selenium version

4.0.0

What are the browser(s) and version(s) where you see this issue?

Chrome96, MSEdge96, MSEdge97

What are the browser driver(s) and version(s) where you see this issue?

chromedriverVersion: '96.0.4664.35, msedgedriverVersion: 97.0.1060.2

Are you using Selenium Grid?

Na

@github-actions
Copy link

github-actions bot commented Nov 9, 2021

@fenchu, thank you for creating this issue. We will troubleshoot it as soon as we can.


Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

@AutomatedTester
Copy link
Member

I have tried to recreate it, I don't have chrome 96, but the following worked with Chrome 95.

>>> driver.get("https://www.trk.qa-04.buypass.no/key-registration")
>>> el = driver.find_element(By.XPATH, "//bp-button[@text='Buypass ID']")
>>> shadow_root = driver.execute_script(f"return arguments[0].shadowRoot", el)
>>> shadow_root
<selenium.webdriver.remote.webelement.WebElement (session="a46ecc2a0b50436187cb24175222129a", element="f6839afa-c89f-449d-8892-943c1e38bf06")>
>>> shadow_root.find_element(By.CSS_SELECTOR, "button").click()
>>> driver.quit()

Please log chromedriver issues with the chromedriver project.

https://sites.google.com/a/chromium.org/chromedriver/help

Please be sure to include a completely reproducible test script for them, without a way to reproduce the issue you are seeing there is no good way for them to fix the problem. A reproducible test script includes a WebDriver script and a link to the page in question. If you can not provide a link, create one using jsfiddle.net

@AutomatedTester
Copy link
Member

please can you link to the issue when you have raised it in the chromium bug tracker

@fenchu
Copy link
Author

fenchu commented Nov 10, 2021

Seems it is the chromedriver v96 only, I can run chrome v96 with chromedriver v95 and it works fine. submittet chromium bug: https://bugs.chromium.org/p/chromedriver/issues/detail?id=3948

@fenchu fenchu changed the title [馃悰 Bug]: driver.execute_script() does not work after chrome/msedge v96 update - return dict and not WebDriver object [馃悰 Bug]: driver.execute_script() does not work with chromedriver/msedgedriver v96 - return dict and not WebDriver object Nov 10, 2021
@christian-bromann
Copy link
Contributor

This doesn't seem like a Chromedriver bug for me. It returns the expected response: a shadow element reference, e.g.:

INFO webdriver: COMMAND executeScript("return ((elem) => elem.shadowRoot).apply(null, arguments)", <object>)                                   
INFO webdriver: [POST] http://localhost:4444/session/7f804512ec08c2a736eea534fa85ec0e/execute/sync
INFO webdriver: DATA {
  script: 'return ((elem) => elem.shadowRoot).apply(null, arguments)',
  args: [
    {
      'element-6066-11e4-a52e-4f735466cecf': 'a6aec538-f29b-4a44-9e95-5e647eaf1f8e'
    }
  ]  
}    
INFO webdriver: RESULT {
  'shadow-6066-11e4-a52e-4f735466cecf': 'aa88a7dd-158e-4b5d-9111-1b19397971d5'                                         
}

I just implemented the shadow commands defined in the spec in this WebdriverIO PR and it allows me to fetch shadow elements using getElementShadowRoot or execute, e.g.:

// open example page
await browser.url('chrome://downloads')

const element = await browser.findElement('tag name', 'downloads-manager')

// get shadow root either using execute command
const shadowRoot = await browser.execute(
    (elem) => elem.shadowRoot, element)

// or get shadow root with the protocol command
const shadowRoot = await browser.getElementShadowRoot(
     element['element-6066-11e4-a52e-4f735466cecf']
)

// fetch element within that shadow root
const elementRef = await browser.findElementFromShadowRoot(
    shadowRoot['shadow-6066-11e4-a52e-4f735466cecf'],
    'css selector',
    '#no-downloads span'
)

console.log(await browser.$(elementRef).getText());
// returns "Files you download appear here"

@AutomatedTester
Copy link
Member

Thanks for that @christian-bromann , we realised that in the TLC meeting yesterday and I will be getting this sorted for 4.1

@fenchu
Copy link
Author

fenchu commented Nov 16, 2021

So there is no workaround until this is fixed in selenium 4.1?,
hopefully newer chrome can support v95.0.4638.17 until then.

@phylor
Copy link

phylor commented Nov 17, 2021

Can confirm that the problem exists with Chrome v96, but not in Chrome v95.

If you happen to be on a Debian based system, the following might work to downgrade Chrome:

CHROME_VERSION=95.0.4638.69-1
wget --no-verbose -O /tmp/chrome.deb https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_${CHROME_VERSION}_amd64.deb
sudo apt remove -y google-chrome-stable
sudo apt install -y /tmp/chrome.deb

@derRichter
Copy link

we can confirm this problem with chromedriver 96, all fine with driver 95.x on newest chrome-browser 96.x
https://bugs.chromium.org/p/chromedriver/issues/detail?id=3948
we use selenium version 3.141.59, hope there is no force to use selenium 4?

my workaround at the moment is to use or downgrade to version 95.0.4638.69.
works also fine with chromium/chrome browser 96.x

good on this constellation is that we got no (error-)message like
"This version of ChromeDriver only supports Chrome version..." or
"Chrome version must be ..."

hope the next stable of chromium-browser 97 is also compatible to chromedriver 95 or
the other way round

@AutomatedTester
Copy link
Member

Fixed in 0c10993 for python.

@github-actions
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked and limited conversation to collaborators Dec 19, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants