Screenshots are not captured if client.end() is not called right after failure.

[LucaLis]

Description

I've noticed that in certain cases screenshots are not captured - it seems that it happens when nightwatch is configured using:

end_session_on_fail: false,
skip_testcases_on_fail: false,

and client.end() is not called directly after failure (e.g. in after hook).

Sample test

Here is an example of such issue. In the console log details below you will notice that there is no trace of saveScreenshot and screenshot commands (eg. → Running command: saveScreenshot). If you comment out test named 'Dummy test' and rerun - screenshot will be captured and relavant commands will be present in log.

sampleTest.js

module.exports = {
    '@tags': ['screenshot'],
  
    'This test should fail and capture screenshot': function(client) {
      client.url('http://google.com');
      client.waitForElementPresent('#thisWontBePresent')
    },

// comment out dummy test below and rerun the test to see that in such setup screens are captured 
    'Dummy test': function(client) {
      client.pause(100)
    },
        
    after(client) {
      client.end();
    }
  };

nightwatch.conf.js

const chromedriver = require('chromedriver');

module.exports = {
  src_folders: ['test'],
  silent: false,

  end_session_on_fail: false,
  skip_testcases_on_fail: false,

  globals:{
    waitForConditionTimeout: 1000
  },

  test_settings:{
    default:{
      screenshots: {
        enabled: true,
        path: 'screenshots',
        on_failure: true,
        on_error: true
      },

      webdriver: {
        start_process: true,
        port: 9515,
        server_path: chromedriver.path,
      },
    
      desiredCapabilities : {
        browserName : 'chrome',
        chromeOptions: {
          w3c: false
        }
      },
    }
  }
};

Verbose output:

console log - with dummy test - screenshots not captured

PS C:\dev\nightwatch-website-tests> npm run test -- --tag=screenshot

> nightwatch-website-tests@0.0.1 test C:\dev\nightwatch-website-tests
> nightwatch "--tag=screenshot"

 Starting ChromeDriver on port 9515...
 ChromeDriver up and running on port 9515 with pid: 1144 (131ms).

[Issues\screenshots] Test Suite
===============================
- Connecting to localhost on port 9515...
   Request POST  /session  
   { desiredCapabilities:
      { browserName: 'chrome',
        platform: 'ANY',
        chromeOptions: { w3c: false },
| Connecting to localhost on port 9515...

DevTools listening on ws://127.0.0.1:55083/devtools/browser/70c8323a-cdcb-4335-869f-495236fcc727
- Connecting to localhost on port 9515...
   Response 200 POST /session (2733ms)
   { sessionId: '66380b1e2571a9b83f05041fd21bc9bb',
     status: 0,
     value:
      { acceptInsecureCerts: false,
        acceptSslCerts: false,
        applicationCacheEnabled: false,
        browserConnectionEnabled: false,
        browserName: 'chrome',
        chrome:
         { chromedriverVersion:
            '81.0.4044.69 (6813546031a4bc83f717a2ef7cd4ac6ec1199132-refs/branch-heads/4044@{#776})',
           userDataDir:
            'C:\\Users\\llis\\AppData\\Local\\Temp\\scoped_dir1144_423881939' },
        cssSelectorsEnabled: true,
        databaseEnabled: false,
        'goog:chromeOptions': { debuggerAddress: 'localhost:55083' },
        handlesAlerts: true,
        hasTouchScreen: false,
        javascriptEnabled: true,
        locationContextEnabled: true,
        mobileEmulationEnabled: false,
        nativeEvents: true,
        networkConnectionEnabled: false,
        pageLoadStrategy: 'normal',
        platform: 'Windows',
        proxy: {},
        rotatable: false,
        setWindowRect: true,
        strictFileInteractability: false,
        takesHeapSnapshot: true,
        takesScreenshot: true,
        timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
        unexpectedAlertBehaviour: 'ignore',
        version: '81.0.4044.129',
        webStorageEnabled: true,
i Connected to localhost on port 9515 (2797ms).
  Using: chrome (81.0.4044.129) on Windows platform.

 Received session with ID: 66380b1e2571a9b83f05041fd21bc9bb

 → Running [before]:
 → Completed [before].
Running:  This test should fail and capture screenshot

 → Running [beforeEach]:
 → Completed [beforeEach].

 → Running command: url ('http://google.com')
   Request POST  /session/66380b1e2571a9b83f05041fd21bc9bb/url  
   { url: 'http://google.com' }
   Response 200 POST /session/66380b1e2571a9b83f05041fd21bc9bb/url (958ms)
   { sessionId: '66380b1e2571a9b83f05041fd21bc9bb',
     status: 0,
  → Completed command: url ('http://google.com') (970ms)

 → Running command: waitForElementPresent ('#thisWontBePresent')
   Request POST  /session/66380b1e2571a9b83f05041fd21bc9bb/elements  
   { using: 'css selector', value: '#thisWontBePresent' }
   Response 200 POST /session/66380b1e2571a9b83f05041fd21bc9bb/elements (22ms)
   { sessionId: '66380b1e2571a9b83f05041fd21bc9bb',
     status: 0,
     value: [] }
   Request POST  /session/66380b1e2571a9b83f05041fd21bc9bb/elements  
   { using: 'css selector', value: '#thisWontBePresent' }
   Response 200 POST /session/66380b1e2571a9b83f05041fd21bc9bb/elements (22ms)
   { sessionId: '66380b1e2571a9b83f05041fd21bc9bb',
     status: 0,
     value: [] }
   Request POST  /session/66380b1e2571a9b83f05041fd21bc9bb/elements  
   { using: 'css selector', value: '#thisWontBePresent' }
   Response 200 POST /session/66380b1e2571a9b83f05041fd21bc9bb/elements (23ms)
   { sessionId: '66380b1e2571a9b83f05041fd21bc9bb',
     status: 0,
     value: [] }
× Timed out while waiting for element <#thisWontBePresent> to be present for 1000 milliseconds. - expected "found" but got: "not found" (1100ms)
    at Object.This test should fail and capture screenshot (C:\dev\nightwatch-website-tests\test\issues\screenshots.js:6:14)
    at process._tickCallback (internal/process/next_tick.js:68:7) 

  → Completed command: waitForElementPresent ('#thisWontBePresent') (1111ms)
 → Running [afterEach]:
 → Completed [afterEach].

FAILED: 1 assertions failed (2.098s)
Running:  Dummy test

 → Running [beforeEach]:
 → Completed [beforeEach].

 → Running command: pause (100)
  → Completed command: pause (100) (102ms)
 → Running [afterEach]:
 → Completed [afterEach].
No assertions ran.

 → Running [after]:

 → Running command: end ()

 → Running command: session ('delete', [Function])
   Request DELETE  /session/66380b1e2571a9b83f05041fd21bc9bb  
   Response 200 DELETE /session/66380b1e2571a9b83f05041fd21bc9bb (34ms)
   { sessionId: '66380b1e2571a9b83f05041fd21bc9bb',
     status: 0,
     value: null }
  → Completed command: end () (76ms)
  → Completed command: session ('delete', [Function]) (64ms)
 → Completed [after].
_________________________________________________

TEST FAILURE:  1 assertions failed, 0 passed (5.375s)

 × issues\screenshots
 – This test should fail and capture screenshot (2.098s)
   Timed out while waiting for element <#thisWontBePresent> to be present for 1000 milliseconds. - expected "found" but got: "not found" (1100ms)
       at Object.This test should fail and capture screenshot (C:\dev\nightwatch-website-tests\test\issues\screenshots.js:6:14)
       at process._tickCallback (internal/process/next_tick.js:68:7)

 Wrote report file to: tests_output\issues\CHROME_81.0.4044.129_Windows_screenshots.xml.
 Wrote log file to: C:\dev\nightwatch-website-tests\chromedriver.log.
 ChromeDriver process closed.
npm ERR! code ELIFECYCLE
npm ERR! errno 5
npm ERR! nightwatch-website-tests@0.0.1 test: `nightwatch "--tag=screenshot"`
npm ERR! Exit status 5
npm ERR!
npm ERR! Failed at the nightwatch-website-tests@0.0.1 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\llis\AppData\Roaming\npm-cache\_logs\2020-05-06T21_06_57_248Z-debug.log

console log - dummy test commented out - screenshots captured

PS C:\dev\nightwatch-website-tests> npm run test -- --tag=screenshot

> nightwatch-website-tests@0.0.1 test C:\dev\nightwatch-website-tests
> nightwatch "--tag=screenshot"

 Starting ChromeDriver on port 9515...
 ChromeDriver up and running on port 9515 with pid: 12684 (128ms).

[Issues\screenshots] Test Suite
===============================
- Connecting to localhost on port 9515...
   Request POST  /session  
   { desiredCapabilities:
      { browserName: 'chrome',
        platform: 'ANY',
        chromeOptions: { w3c: false },
| Connecting to localhost on port 9515...

DevTools listening on ws://127.0.0.1:55117/devtools/browser/45c2758a-c0d1-4143-b583-13b277f4efa3
- Connecting to localhost on port 9515...
   Response 200 POST /session (2692ms)
   { sessionId: '9d05dfc50187f0152a8a485724b6678d',
     status: 0,
     value:
      { acceptInsecureCerts: false,
        acceptSslCerts: false,
        applicationCacheEnabled: false,
        browserConnectionEnabled: false,
        browserName: 'chrome',
        chrome:
         { chromedriverVersion:
            '81.0.4044.69 (6813546031a4bc83f717a2ef7cd4ac6ec1199132-refs/branch-heads/4044@{#776})',
           userDataDir:
            'C:\\Users\\llis\\AppData\\Local\\Temp\\scoped_dir12684_1085042448' },
        cssSelectorsEnabled: true,
        databaseEnabled: false,
        'goog:chromeOptions': { debuggerAddress: 'localhost:55117' },
        handlesAlerts: true,
        hasTouchScreen: false,
        javascriptEnabled: true,
        locationContextEnabled: true,
        mobileEmulationEnabled: false,
        nativeEvents: true,
        networkConnectionEnabled: false,
        pageLoadStrategy: 'normal',
        platform: 'Windows',
        proxy: {},
        rotatable: false,
        setWindowRect: true,
        strictFileInteractability: false,
        takesHeapSnapshot: true,
        takesScreenshot: true,
        timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
        unexpectedAlertBehaviour: 'ignore',
        version: '81.0.4044.129',
        webStorageEnabled: true,
i Connected to localhost on port 9515 (2748ms).
  Using: chrome (81.0.4044.129) on Windows platform.

 Received session with ID: 9d05dfc50187f0152a8a485724b6678d

 → Running [before]:
 → Completed [before].
Running:  This test should fail and capture screenshot

 → Running [beforeEach]:
 → Completed [beforeEach].

 → Running command: url ('http://google.com')
   Request POST  /session/9d05dfc50187f0152a8a485724b6678d/url  
   { url: 'http://google.com' }
   Response 200 POST /session/9d05dfc50187f0152a8a485724b6678d/url (981ms)
   { sessionId: '9d05dfc50187f0152a8a485724b6678d',
     status: 0,
     value: null }
  → Completed command: url ('http://google.com') (994ms)

 → Running command: waitForElementPresent ('#thisWontBePresent')
   { using: 'css selector', value: '#thisWontBePresent' }
   Response 200 POST /session/9d05dfc50187f0152a8a485724b6678d/elements (11ms)
   { sessionId: '9d05dfc50187f0152a8a485724b6678d',
     status: 0,
     value: [] }
   Request POST  /session/9d05dfc50187f0152a8a485724b6678d/elements  
   { using: 'css selector', value: '#thisWontBePresent' }
   Response 200 POST /session/9d05dfc50187f0152a8a485724b6678d/elements (22ms)
   { sessionId: '9d05dfc50187f0152a8a485724b6678d',
     status: 0,
     value: [] }
   Request POST  /session/9d05dfc50187f0152a8a485724b6678d/elements  
   { using: 'css selector', value: '#thisWontBePresent' }
   Response 200 POST /session/9d05dfc50187f0152a8a485724b6678d/elements (25ms)
   { sessionId: '9d05dfc50187f0152a8a485724b6678d',
     status: 0,
     value: [] }
× Timed out while waiting for element <#thisWontBePresent> to be present for 1000 milliseconds. - expected "found" but got: "not found" (1081ms)
    at Object.This test should fail and capture screenshot (C:\dev\nightwatch-website-tests\test\issues\screenshots.js:6:14)
    at process._tickCallback (internal/process/next_tick.js:68:7) 

  → Completed command: waitForElementPresent ('#thisWontBePresent') (1085ms)
 → Running [afterEach]:
 → Completed [afterEach].

FAILED: 1 assertions failed (2.087s)
 → Running [after]:

 → Running command: end ()
 Failures in "This test should fail and capture screenshot". Taking screenshot...

 → Running command: saveScreenshot ('C:\dev\nightwatch-website-tests\screenshots\issues\screenshots\This-test-should-fail-and-capture-screenshot_FAILED_May-06-2020-231317-GMT+0200.png', [Function])

 → Running command: screenshot (false, [Function])
   Request GET  /session/9d05dfc50187f0152a8a485724b6678d/screenshot  
   Response 200 GET /session/9d05dfc50187f0152a8a485724b6678d/screenshot (292ms)
   { sessionId: '9d05dfc50187f0152a8a485724b6678d',
     status: 0,
     value: '',
     suppressBase64Data: true }
  → Completed command: screenshot (false, [Function]) (316ms)
  → Completed command: saveScreenshot ('C:\dev\nightwatch-website-tests\screenshots\issues\screenshots\This-test-should-fail-and-capture-screenshot_FAILED_May-06-2020-231317-GMT+0200.png', [Function]) (323ms)

 → Running command: session ('delete', [Function])
   Request DELETE  /session/9d05dfc50187f0152a8a485724b6678d  
   Response 200 DELETE /session/9d05dfc50187f0152a8a485724b6678d (30ms)
   { sessionId: '9d05dfc50187f0152a8a485724b6678d',
     status: 0,
     value: null }
  → Completed command: end () (390ms)
  → Completed command: session ('delete', [Function]) (42ms)
 → Completed [after].
_________________________________________________

TEST FAILURE:  1 assertions failed, 0 passed (5.494s)

 × issues\screenshots
 – This test should fail and capture screenshot (2.087s)
   Timed out while waiting for element <#thisWontBePresent> to be present for 1000 milliseconds. - expected "found" but got: "not found" (1081ms)
       at Object.This test should fail and capture screenshot (C:\dev\nightwatch-website-tests\test\issues\screenshots.js:6:14)
       at process._tickCallback (internal/process/next_tick.js:68:7)

 Wrote report file to: tests_output\issues\CHROME_81.0.4044.129_Windows_screenshots.xml.
 Wrote log file to: C:\dev\nightwatch-website-tests\chromedriver.log.
 ChromeDriver process closed.
npm ERR! code ELIFECYCLE
npm ERR! errno 5
npm ERR! nightwatch-website-tests@0.0.1 test: `nightwatch "--tag=screenshot"`
npm ERR! Exit status 5
npm ERR!
npm ERR! Failed at the nightwatch-website-tests@0.0.1 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\llis\AppData\Roaming\npm-cache\_logs\2020-05-06T21_13_18_430Z-debug.log

Your Environment

Executable	Version
nightwatch --version	1.3.4
npm --version	6.9.0
node --version	v10.16.3

Browser driver	Version
NAME	VERSION
chromedriver	81.0.4044.69

OS	Version
NAME	VERSION
Windows 10	1607

[Pieras2]

I don't know why but in my case I don't get screenshots - folder is not created.
Some time it worked :/

[Pieras2]

screenshot is not made even on_error: true, if end_session_on_fail: false also doesn't help

[ajpetersons]

Would be nice if this issue was addressed. I can find many threads that mention taking a screenshot after an assertion failure. In our case we want to continue running the script even if some commands fail (even if those are element commands failing) but have a screenshot of the failure. I'm afraid I'm not able to implement a solution that will properly work for all cases. It would be very much appreciated if this issue was fixed.

[Pieras2]

@beatfactor can we have some help? My tests fail on errors (Azure DevOps) and I cannot reproduce the problems on my desk - without screenshot (NW 1.4.3) I don't know what happens. Is there any possible workaround?

EDIT:

• will try what mentioned LucaLis

[Pieras2]

If inside the code, after the line with browser.verify I add browser.end(), yes as @LucaLis says the screenshot is made, but in case the code goes on with another lines of code, even the browser.end() is in after() hook, the screenshot is not made. In my case after is used as async because insidde I call a method that returns a promise.

[gap777]

We're experiencing something like this all the time. We run our tests on semaphoreCI 2.0, and half the time they fail we don't get screenshots. It makes for such a frustrating experience to have a flaky test runner. What has to happen to fix this? @beatfactor Can you advise? This is becoming a big enough problem to cause our team to evaluate alternatives like cypress.io.

[Pieras2]

Unfortunately even if screenshot is made it is made only for the last assertion in the test suite :(
See #2645 < this should be rather a bug, but ok.

[beatfactor]

If end_session_on_fail is turned off than .end() will not be called and screenshots will not be taken. But the test execution is still halted and you would normally still need to close the session yourself in order to avoid having browsers open.

So I would say that there's no benefit in turning off end_session_on_fail, unless you would want to do other things beside taking the screenshot. The only reason for even adding this option in the first place was so that users can take screenshots before the session was closed, but now the functionality is there.

But I can see that this can lead to many confusions so I think we should take the screenshot regardless of the end_session_on_fail state and maybe even take the screenshot for each assertion failure.

[Pieras2]

That would be really great ❤