Understanding the Command Queue

When Nightwatch runs a test, it processes its commands in a list known as the command queue. This list manages the asynchronous execution of the commands defined in that test.

As a queue, the command queue generally follows the rule of first in, first out (FIFO). The first command you call in a test is the first executed when the test runs. The second command is called next, followed by the next command up until the last command added, which becomes the last command executed.

Command Queue Creation

The command API in Nightwatch - accessible via the object passed into test cases, usually called "client" or "browser" - consists of a collection of methods that are used to construct the command queue. When you call a Nightwatch command such as click(), you're not sending the Selenium client the command to immediately click something, at least not right away. That method instead adds a "click" command to the command queue. After the test case function has resolved, something it does synchronously (commands are non-blocking), it traverses the command queue as defined by the Nightwatch commands you called, running through the queue executing each command in it asynchronously.

Consider the following test:

// test with 'click button' test case

module.exports = {
  'click button': function (browser) {
    browser
      .url('example.com')
      .click('button')
      .end();
  }
};

Here, the 'click button' test case calls three methods from the browser object: url(), click(), and end(). When Nightwatch runs this test, it calls the 'click button' function which, in turn, calls each of these methods. These methods run synchronously, one directly and immediately after the other, and populates the command queue without performing any Selenium tasks. The result is a queue that looks something like:

// 'click button' test case command queue

[
  {command: 'url', args: ['example.com']},
  {command: 'click', args: ['button']},
  {command: 'end', args: []}
]

Its only after the test case function resolves that the queue is traversed and the commands executed, either in the context of Selenium, or through whatever other implementation they may have.

Command Queue Execution

Most commands in Nightwatch are Selenium commands, wrapping a single, or two or more Selenium requests. These are handled through the WebDriver API which consists of a series of HTTP endpoints to perfom actions on a WebDriver client, such as a Selenium Server instance. The HTTP requests made by Nightwatch are non-blocking with the results of the requests handled in callbacks. The click command, for example, in the deep internals of Nightwatch code is executed using something like:

// running the Selenium click command

http.request({
    method: 'POST',
    path: '/session/1/element/2/click'
  }, function (response) {
    // request complete
});

The non-blocking, asynchronous nature of Selenium-based commands, or even other asynchronous commands like pause() (effectively a command wrapper for setTimeout()) means that for operations like this to run in sequence, they must wait for the completion of the previous. The succinct, chainable command API of Nightwatch doesn't force this requirement on you, instead handling it internally through the inner workings of the command queue. It acts as a cache saving off the operations you wish to perform until after the test function completes at which point, it runs them each in sequence asynchronously.

// command queue execution pseudo code

commandQueue = []

runTestCaseFunction(browser) // synchronous (adds to commandQueue)

for command of commandQueue {
   await runCommand(command) // asynchronous (commands wait for previous to finish)
}

Command Callbacks and the Dynamic Queue

Just about every native Nightwatch command supports a callback function to be called when the command has been executed within the command queue. These callbacks, as with the execution of the commands themselves within the queue, are called asynchronously.

// click command callback

browser.click('button', function (result) {
  // when the queue has finished executing the click command
  // where `result` is the response from the Selenium click operation
});

The primary purpose for callbacks is to allow you to capture data from the test while its running, but they also provide an additional opportunity to add commands to the command queue. Unlike the command methods called directly within the test case function body, command methods called in a callback get added to the current location of the command queue within its traversal. Commands added in the click() callback above, for example, would be added directly after the click command's location within the queue rather than being added to the end (the end being the standard FIFO enqueue behavior seen with commands outside of callbacks). Consider a revised sample test:

// test with 'click button' test case

module.exports = {
  'click button': function (browser) {
    browser
      .url('example.com')
      .click('button', function (result) {
        browser.pause(100);
      })
      .end();
  }
};

When run, initially the command queue starts as:

// 'click button' test case command queue after test case called

[
  {command: 'url', args: ['example.com']},
  {command: 'click', args: ['button']},
  {command: 'end', args: []}
]

Note that there is no pause yet, because the pause() command method doesn't get called until after the click command is executed when the queue is being processed. This is the state of the queue immediately after the test case function runs.

When the queue starts, it runs the url command, waits for it to finish, then runs the click command which then calls the callback. When the callback runs, it adds the pause command to the queue, giving us:

// 'click button' test case command queue after click callback called

[
  {command: 'url', args: ['example.com']}, // executed
  {command: 'click', args: ['button']}, // executed
  {command: 'pause', args: [100]}, // <-- just added
  {command: 'end', args: []}
]

When the queue continues to the next step, it then runs the newly added pause command - as that is now the next command in the queue - before eventually running the final end command completing the test case.

One thing to realize is that though the pause command was added out of order, asynchronously, the order in which you read each of the commands in the source code is ultimately the same order they are executed when the command queue is traversed. The added order is url, click, end, pause; while the executed order - and the order read in the source - is url, click, pause, end. This makes it easy to look at source code and immediately get a general idea of the order of commands ultimately getting run.

Working with Data in the Queue

As mentioned earlier, callbacks provide a way to get data from a running test such as values of textfields, attributes, etc. This data isn't available when the test function runs because the actual running of the test case (traversing of the command queue) doesn't start until after the test case function has already run to completion.

// capturing test data from callback

var text;
browser.getValue('#input', function (result) {
  text = result.value; // captured during command queue execution
});

Similar to how commands in callbacks aren't in the command queue right away, values captured this way also aren't available until the test is running. In a callback, all code directly in the test case function body has already resolved, and the only place any other code will run is in other callbacks. This is important to keep in mind because it can be easy to think this might work:

// incorrect usage of a callback value

var text;
browser
  .getValue('#input', function (result) {
    text = result.value;
  })
  .setValue('#output', text); // WRONG: text is undefined

The problem here is that the setValue() call happens in the main test case function call, before the callback is called when text is still undefined. For setValue() to have the correct value for text, it must be called within, or some time after, the getText() callback:

// correct usage of a callback value

var text;
browser.getValue('#input', function (result) {
  text = result.value;
  browser.setValue('#output', text); // RIGHT
});

Any additional dependencies on text may also go in the getValue() callback, and any command there may have their own callbacks which may include additional commands ultimately giving you a mess of nested callbacks.

// nested callbacks

var text, text2, text3; // ...
browser.getValue('#input', function (result) {
  text = result.value;
  browser.getValue('#' + text, function (result) {
    text2 = result.value;
    browser.getValue('#' + text2, function (result) {
      text3 = result.value;
      browser.getValue('#' + text3, function (result) {
        // ...
      });
    });
  });
});

To counter this, you can use the perform() command.

The perform() Command

Nightwatch's perform() command is effectively a no op command that exists only to provide a callback allowing you to run code within the context of the running command queue. Within the callback, you're ensured to have code which will run after any callbacks defined above it, even if nested.

// perform to help combat nested callbacks

var text, text2, text3;
browser
  .getValue('#input', function (result) {
    text = result.value;
    browser.getValue('#' + text, function (result) {
      text2 = result.value;
    });
  })
  .perform(function () {
    browser.getValue('#' + text2, function (result) {
      text3 = result.value;
      browser.getValue('#' + text3, function (result) {
        // ...
      });
    });
  });

Let's rundown of how this set of commands would play out in a test case, starting from when the test case is first called:

// initial test case queue

[
  {command: 'getValue', args: ['#input', callback]},
  {command: 'perform', args: [callback]},
]

// queue after first getValue ('#input') callback

[
  {command: 'getValue', args: ['#input', callback]}, // executed, text defined
  {command: 'getValue', args: ['#text', callback]}, // <-- added
  {command: 'perform', args: [callback]},
]

// queue after second getValue ('#text') callback

[
  {command: 'getValue', args: ['#input', callback]}, // executed, text defined
  {command: 'getValue', args: ['#text', callback]}, // executed, text2 defined
  {command: 'perform', args: [callback]},
]

// second getValue callback adds no commands, perform is next

[
  {command: 'getValue', args: ['#input', callback]}, // executed, text defined
  {command: 'getValue', args: ['#text', callback]}, // executed, text2 defined
  {command: 'perform', args: [callback]}, // executed
  {command: 'getValue', args: ['#text2', callback]}, // <-- added
]

// final queue after third getValue ('#text2') callback

[
  {command: 'getValue', args: ['#input', callback]}, // executed, text defined
  {command: 'getValue', args: ['#text', callback]}, // executed, text2 defined
  {command: 'perform', args: [callback]}, // executed
  {command: 'getValue', args: ['#text2', callback]}, // executed, text3 defined
  {command: 'getValue', args: ['#text3', callback]}, // <-- added
]

Callbacks (inline), like the commands themselves, are executed in the same order they're read in the code. So even though the perform() command itself is added to the queue before the text2 defining getValue() call is, its callback doesn't execute until after text2 has been defined in that getValue()'s callback.

Setting a breakpoint to inspect a page in browser

Sometimes it is important to inspect a page in browser in the middle of a test case run. For example, to verify used selectors. To pause execution at the right moment set a breakpoint inside a callback:

browser.perform(function () {
    console.log('dummy statement'); // install a breakpoint here
  });

Custom Commands in the Command Queue

When you create a custom Nightwatch command, either in its function or class-based form, the code in that command is automatically deferred and executed during command queue traversal rather than being called when the command function is called in the test code. This acts as if the code is automatically wrapped in a perform().

// custom command module myCustomCommand.js

module.exports = function myCustomCommand () {
  console.log('Custom command'); // logged during queue traversal, not when called
};

// test using custom command

module.exports = {
  'custom command': function (browser) {
    console.log('Calling custom command');
    browser.myCustomCommand();
    console.log('Called custom command');
};

// Output:
//-> Calling custom command
//-> Called custom command
//-> Custom command

Notice that the console.log() calls in the test case get called immediately when the test case function is run, while the custom command console.log() isn't seen until after, when the custom command is run within the command queue.

Page Object Commands in the Command Queue

Page object commands do not share the queuing behavior of custom commands. Their code is not automatically added to the queue, and is instead run immediately when called. If you'd like custom page object commands to be run in the command queue, you should wrap them in a perform() command.

// page object module myPageObject.js

module.exports = {
  elements: {},
  commands: [
    {
      myCommand: function () {
        console.log('Page object command as method'); // called immediately
      },
      myQueuedCommand: function () {
        this.api.perform(function () {
          console.log('Page object command as queued command'); // called in queue
        }.bind(this)); // ensure `this` in perform is still page object
      }
    }
  ]
};

// test using page object commands

module.exports = {
  'page object commands': function (browser) {
    var myPageObject = browser.page.myPageObject();

    console.log('Calling page object commands');
    myPageObject.myCommand();
    myPageObject.myQueuedCommand();
    console.log('Called page object commands');
  }
};

// Output:
//-> Calling page object commands
//-> Page object command as method
//-> Called page object commands
//-> Page object command as queued command

Here, you can see that the myCommand() page object command's console.log() is called immediately when the function is called, putting its output between the two lines of output in the test case, unlike the results seen with normal custom commands. The other hand, the page object command myQueuedCommand() exhibits the custom command behavior because it was wrapped in a perform(), the code then being called during command queue traversal.

Note: if your page object command is only calling other commands which are, themselves, queued, there is no need to wrap them in a perform().