Unable to automate login to gmail via TagUI - Google now unblock?

[LIONEL743]

Hi Ken,

Recently there seems to be a change in chrome's behavior where the chrome instance created by TagUI cannot be used to login to Gmail. This seems to have started early this week. I have tried both normal and incognito mode but it seems like both encountered the same issue. The error returned by Gmail after keying in the username and password goes like "This browser or app may not be secure." (couldn't attach the screenshot via GitHub, seems to get an error on this)

Tried on different PCs and different accounts, the result were the same as well. However, using just chrome without initiating from TagUI on these PCs to login gmail works fine. Do you experience the same issue too?

Thanks for your help!

Regards,
Lionel

[LIONEL743]

Attached a snippit on the login codes below :

https://accounts.google.com/ServiceLogin
wait 2 seconds
type email as test@gmail.com
click identifierNext
type password as passwordstr
click passwordNext

[kensoh]

Hi Lionel, thanks for raising this!

Yes I can replicate the issue. After getting the message I click for more info and see the following screenshot (to attach files, just drag and drop into this comment textbox, but some company network maybe has firewall that blocks uploading to GitHub this way) -

It looks like Chrome has an update that explicitly prevents automated browser activities, see point 3 above on 'Use automation testing frameworks' being one of the scenarios where logins can be blocked. If Google explicitly wants to block, there may not be a good solution for this. Copying @siowyisheng to tap on your ideas and experience on how to tackle this.

In the meantime, a viable alternative is if your automation can be done using visual automation (see here for details https://github.com/kelaberetiv/TagUI#visual-automation), then you can use the computer vision, OCR, keyboard and mouse automation capabilities of TagUI to automate the same workflow on your real Chrome browser with your real user login.

[kensoh]

Adding on, looks like many apps and users are impacted and many are unhappy with the change -
https://support.google.com/accounts/thread/22873505

[siowyisheng]

Hm I don't know of any good way to get around this, even if we want to (considering that they are intentionally blocking automated web browser use).

[kensoh]

We might have to follow how other test automation tools like Selenium and especially Puppeteer are going to address this. Problem with Chrome is it doesn't allow downloading of old releases.

I googled and saw that a workaround is downloading older versions of Chromium. But that will almost certainly requires modification of TagUI config to work - in tagui\src\tagui and tagui.cmd. Would not want to go via this path as apps are still being tested for Chrome, not Chromium.

So using Chromium will not exactly replicate the same webpage rendering and experience as a normal user would expect, and that will add another layer of complexity (and thus potential point of failure for automations).

[kensoh]

I'll follow-up on this one

[kensoh]

Some findings -

1. Puppeteer, besides using --remote-debugging-port to connect Chrome, it uses --remote-debugging-pipe, in below Launcher module.

https://github.com/puppeteer/puppeteer/blob/8b49dc62a62282543ead43541316e23d3450ff3c/lib/Launcher.js

2. From this issue at chrome-remote-interface (normally how Node.js folks and app-makers will use to connect to Chrome, if not using Puppeteer) - --remote-debugging-pipe support cyrus-and/chrome-remote-interface#381, it looks like in v2, they now defaults to using --remote-debugging-pipe instead of --remote-debugging-port. I added a comment there to share findings and follow any new developments there at chrome-remote-interface.

From above, it could be now that Puppeteer switches to using the pipe mode (using the file-system) versus port mode (using memory) to do the communication with Chrome, it leaves Chrome flexible to block out apps that uses port mode to communicate and flag as automated browser.

Of course, this is a hypothesis, which can be verified if we do some POC to connect to Chrome with pipe instead of port. I don't know how to do that yet though. But putting this info here in case we want to review switching to pipe mode if that's the best practice Puppeteer is using.

And if the hypothesis is true, it might be the only way to continue logging in automatically to Google accounts and services when users are using TagUI. And we might have to review the technical implementation how to achieve this, and any catch on existing features to switch.

[mbsryu]

If sending mail via Gmail is the issue, would the use of external CLI mailers be a solution for some cases? I've tested the use of SwithMail ( https://sourceforge.net/projects/swithmail/ )with Gmail/G-Suite accounts via their SMTP interface. It works. Has anyone tried Nodemailer ( https://nodemailer.com/about/ ), a node.js mailer module ?

[kensoh]

An update that chrome-remote-interface maintainer responded and he said that using the pipe mode to connect also has this login issue - cyrus-and/chrome-remote-interface#381 (comment)

Thanks @mbsryu yes that should work, though where possible we would prefer to go via UI, so that the automation actions can be seen happening, and also to focus on UI automation instead of through API (though that's possible with api step and API section under Developers Reference) or in cases of CLI, using run step to invoke the target app.

[kensoh]

Seems like Selenium is affected, from one of the users Dumont Reporting at https://support.google.com/accounts/thread/22873505

I'm very upset about this. Has Google blocked all automation scripts? I use a lot of these at work and that would be devastating to my work flow if they have blocked these type of useful bots. My team and I rely on Selenium automation for a lot of our work. Please help!

Another user from above thread Shannon Davis 625 confirmed issue introed in Chrome v79 -

I figured out how to fix my "This browser or app may not be secure" can't log in issue!!!!
I deleted Chrome app and installed Chrome 78.
I was not able to log into my Google accounts at from any browser including Chrome! Once I deleted Chrome 79 app and went back to the 78 I can log into all my accounts!!!

No other leads for now, pending more developments in other tools or user inputs.

[kensoh]

Another Google userGadi B. also experiencing problem with Selenium automation at https://support.google.com/accounts/thread/22873505

Have enough info to verify that Selenium is impacted, they may not have issues raised yet maybe because no one has raised it yet or they have some other channel like Slack or mailing list which we didn't find.

I'm experiencing exactly the same thing. I'm trying to automate Chrome (version 79) via Selenium, to access Google Data Studio in order to download PDF reports. There are dozens of reports to download hence why I need to create this sort of automation from the first place.

Unlike most of the reported errors in the screenshots of this thread, on my end I'm seeing my email address in the screen of the page saying "This browser or app may not be secure". Clicking the shown email address takes me back to the email address input in the first login page, which has an identical result if I try refreshing the page as the text suggests.

When I attempt inputting the correct password and also any other invalid password, I'm getting the same outcome, the screen from the screenshot below is shown, meaning visually no authentication attempt for the password is even taking place.

Google - You have to fix this serious bug. I'm using Chrome browser to access Google account, how in the world can that browser be considered not secured by you?! (Unless you decided to ban Selenium...)

[kensoh]

Someone suggested overriding the user agent setting of the browser, can try this next.

To resolve the error on my implementation I added a property to change the user agent of the browser window.

https://electronjs.org/docs/api/browser-window#winloadurlurl-options

win.loadURL('https://path/to/your/auth/endpoint', { userAgent: 'Chrome' })

[kensoh]

Modifying useragent fails, below still gets insecured browser message and login is blocked -

chrome_step('Network.setUserAgentOverride', {userAgent: 'Chrome'});
chrome_step('Emulation.setUserAgentOverride', {userAgent: 'Chrome'});
https://mail.google.com/mail/u/0/#inbox
wait 30 seconds
chrome_step('Network.setUserAgentOverride', {userAgent: 'Chrome'});
chrome_step('Emulation.setUserAgentOverride', {userAgent: 'Chrome'});
live

Chrome DevTools API reference -
https://chromedevtools.github.io/devtools-protocol/tot/Emulation/#method-setUserAgentOverride
https://chromedevtools.github.io/devtools-protocol/tot/Network#method-setUserAgentOverride

[kensoh]

Hi @LIONEL743 it could turn out that Google may continue to block automated login to Gmail and has no workaround if going by accessing the web elements of the webpage directly. Have you tried using the visual automation method in my first reply above to see if that works for your use case?

[kensoh]

unassigning for now pending more developments. ideal case is Google nudges and re-enable such use cases, less ideal but still good is a workaround for web-based automation found from other tools. worst case, use visual automation since Google wants to start throwing its weight around to decide what should and should not be allowed by users, reminds me of Apple with its recent new releases making things tougher both for users and developers. CC @siowyisheng

[LIONEL743]

Hi Ken,

Yes I did. Unfortunately, because the chrome window is opened using visual automation, that also means it wont be possible to retrieve contents using xpath (which is part of the automation script which i created to pull information from certain sites, before logging into gmail).

Working around this, I had to split my script into two separate parts, one to retrieve the information and store them somewhere, while the other opens a web-browser using visual automation to login and send the information over.

In the meantime, I have also created a script to send information via an outlook account (as a last resort) should the need arises in the future. In the mean time, I am waiting to see how things goes for now.

[kensoh]

Hi Lionel, thanks for sharing your update! I'm hoping that Chrome team do some form of roll-back for this bad change, or some workaround to continue doing automation using web identifiers.

For the splitting into 2 parts, it should still be doable in the same file, just that the scrapping part you use XPath and so on to automate on the TagUI-invoked Chrome browser sessions, and the second part you using visual automation to invoke Chrome and gmail.

Doing in a single browser would be hard with visual automation for scrapping, even with using keyboard step and clipboard() helper function. As it's not that precise and powerful in selecting web elements like using XPath.

[kensoh]

An update that the Google support staff has locked the support question from further posting -
https://support.google.com/accounts/thread/22873505

[kensoh]

Closing issue for now as there's nothing to be done, but keeping a lookout for more datapoint from other users and other apps, to see if something can be done. In the meantime, using visual automation to invoke a normal Chrome session and automating the actions on Gmail that way should work, as the automated login and interactions are transparent to Gmail that way.

[kensoh]

Hi @LIONEL743 and @siowyisheng want to update that today I tried logging in to Gmail using TagUI automation and it seems to work for me! Maybe they roll-back the unnecessary security measure after hearing feedback from the community.

Not sure it will work for you. I just did a simple workflow and it is able to login to Gmail -

timeout 300
https://mail.google.com/mail/u/0/
live

[kensoh]

Adding on, it seems to be selectively work. For AI Singapore GSuite email account it works, but for my personal email account it still gives the following message -

[sfantu]

The question arises ... what exactly is gmail/google detecting ? And if they "detect" --remote-debugging what else are they detecting.

[LIONEL743]

@kensoh Indeed! I retried my script and it works now, strangely

[kensoh]

Oh great! At least from my tests, it looks like they somewhat did a partial rollback. That at least for some accounts (my GSuite Gmail) it works but for some accounts (eg my personal Gmail) it fails.

[kensoh]

@sfantu yes I believe it is by detecting the remote-debugging switch. Detecting that is enough to sniff an automated Chrome session. Due to the architecture of Chrome, it is from what I know the only way to open a backdoor to control Chrome. Without this switch, there is no WebSocket connection for TagUI to control and read data from Chrome.

For accounts that does not work, the alternative is purely use visual automation methods (UI image snapshots, keyboard combinations and keys, (x,y) coordinates of identifiers) to do the Gmail automation. This way Google can't detect because everything should appear like a normal user to Google web server. There are advanced detection methods like tracking mouse movements to see if it contains the 'noise' inherent in human users but hopefully they don't go that far.

[p-funky]

Hey everyone, I just tried something out that worked for me after several hours of trial and error.

Adding args: ['--disable-web-security', '--user-data-dir', '--allow-running-insecure-content' ] to my config resolved the issue.

PS - I was doing an automation using protractor so you might want to find out how to add this to your config with whatever automation tool you are using.

[kensoh]

Hi @p-funky Yinka, I appreciate that you share this info here!

I've tried and unfortunately the workaround does not work in this case, maybe because TagUI always will open a websocket connection to Chrome with the --remote-debugging-port=9222 switch. This seems to be the only way to control Chrome and read from Chrome but it also seems to be what Google uses to detect automated browsers. Below is the error screenshot if I try to login to my personal Gmail. Corporate GSuite Gmail works.

What's interesting is why Protractor works but not TagUI. Hi @siowyisheng, can you have a look at this? Protractor is a testing library by Angular built on WebDriver.

[p-funky]

Hi @kensoh thanks for your response. After further tests yesterday, I realized what I tried was not why it worked i.e. I removed the args and it was still working. Weird thing is that the email was previously been denied access after sign in. Not sure what exactly changed anymore to be honest. Still investigating it and trying out theories.

Two things I know I did and I want to explore:

1. I was initially logging in through stackoverflow's google login method and then redirecting to gmail afterwards. Perhaps doing this via automation somehow cleared up some blockers/issues.
2. I changed my password because I later realized I had made a silly mistake of pushing code out with my personal email and password (feel free to cast a stone at me if you have never made a mistake).

Perhaps one or more of a combination of the above affected my email and gave me access.

[kensoh]

Hi Yinka, thanks for sharing back these details!

Can you tell me more about the Stack Overflow Google login method you tried?

[p-funky]

@kensoh sure. Coincidentally, I have had someone else try that out as well and after about 24 hours, he can now use his email to login directly without going through stackoverflow.
Using automation:

1. Go to https://stackoverflow.com/users/login
2. Select Log in with Google Strategy
3. Enter Google username and password
4. Login to Stackoverflow
5. Go to https://gmail.com

After doing this consistently for like a whole day(24 hours), try automating your login directly to gmail directly... that's what we both have done.
PS - You might want to continue with the stackoverflow login until you at least get a captcha request as we both went through that phase as well.

[kensoh]

Thank you very much for sharing @p-funky Yinka!

I confirm that steps 1-5 works! Fyi @siowyisheng

[jaisonclarck]

@kensoh
Even after using the Stack Overflow Google login method , I still receive Capatcha before the password page, any solutions ?

[p-funky]

@jaisonclarck manually respond to the captcha request. It should stop after a bit.

[jaisonclarck]

@p-funky

Thanx, I will try that ....but what if there is negative possibility ???
Is there any steady solution?

[p-funky]

@jaisonclarck you saw my reponse here yeah? That's the only steady solution I have found thus far. The whole idea is that after about 24 hours of using the Google login strategy via Stack Overflow, you should be able to login directly without having to go through Stack Overflow anymore. Of course, if you (and hopefully you do) get there, then that's a steady solution.

[jaisonclarck]

@p-funky
thank you sir ,