Integration Testing with nodeJS

logo

Recently more and more front-end tools started to move towards Node.js. Even though the JavaScript community and the number of available tools are expanding rapidly there isn’t that much choice in regard to integration testing. However, a move to NodeJS brought us a few very interesting and promising front-end integration testing tools.

Selenium historically was mostly an objective language-based tool. It was mostly used with Java and quite popular with C. Even though there are many Java-based Selenium wrappers and libraries, there were not that many for Javascript until just recently. I have decided to assess these Javascript integration testing tools, compare the performance, usability, efficiency and stability with Java equivalents and in this blog, I will go through my findings.

Both NightwatchJS and Webdriverio are very similar, both are very quick, easy to configure simplified Selenium wrappers; they also serve the same purpose, thus they have similar functionality. After trying both of the tools it seems that the developers are pushing towards the idea of simple, small, quick and efficient integration testing. It feels like they are telling us that massive, clumsy and inefficient Java Selenium tests are the past and they are changing the direction of integration testing. I have to agree, I like that and it makes a lot of sense, however by moving in the direction of lightweight integration testing you lose some of the important bits that clumsy Java tests have (Abstract page factory, proper page object pattern, parallel running, multithreading etc..).

So what are these tools? What are the main features? Both of them are very easy to configure, easy to run and easy to get a very quick feedback. In addition to that, it significantly simplifies the process of writing automated tests.

So how do these tools stack up against Selenium and other Java Selenium wrappers?

Configuration

There is a massive difference in terms of setup time. Selenium, or any Selenium Java wrapper, requires a significant amount of time and effort to set up. Node.js tools, however, take about 5 minutes to set up. This is a great achievement, you can set up your projects testing framework in minutes. In addition, it fits well with their idea of lightweight testing.

Test Runners

Selenium Java wrappers and libraries have two main test runners – JUnit and TestNG, both have their strengths and weaknesses, however, both of them are very slow in comparison with the popular Javascript tools. Before comparing the speeds we have to take into account that the Nodejs tools introduces an additional step in communication between the tool and Selenium Server, which does take time since every call has to go through the additional HTTP request. Even though, it has an additional communication step, after conducting the same test on several test runners I came to the conclusion that Mocha integration tests run ~25% faster than JUnit or TestNG equivalents. This is a huge difference taking into account the additional communication step.

Speed test results
Speed test results.

Test Writing

This is one of the main strengths of Nodejs testing tools. They simplified test writing to the level that the test can be written in significantly less time and requires less technical knowledge from tester. Tests are much easier to read and maintain too, which makes it possible to write a test and scrap it without any bad feeling that you are wasting time. However, some of the decisions were questionable to me, since the framework introduces callback based test writing which makes it hard to edit or chain more advanced scenarios.

JavaScript code example (keep in mind that this code doesn’t need to define @BeforTest and @AfterTest, so that’s basically all you need for the test):

'Enter text and assert that it was submited' : function (browser) {
   browser
      .url('http://localhost:8080/')
      .waitForElementVisible('body', 1000)
      .setValue('body > section > div > header > input', ['nightwatch', browser.Keys.ENTER])
      .waitForElementVisible("body > section > label",1000)
      .getText("body > section > div > section > label", function(result) {
          this.assert.equal(result.value, "nightwatchX");
      })
}

Java code still needs to include optional but important @BeforeTest and @AfterTest code.  In addition, I have skipped the sleep timeouts which are included in Nightwatch methods, that would make Java code even bulkier.

  @Test
  public void testSpeed2() {
    driver.get("http://localhost:8080/");
    driver.findElement(By.cssSelector("body")).isDisplayed();

    driver.findElement(By.cssSelector("body > section > div > header > input")).sendKeys("nightwatch");
    driver.findElement(By.cssSelector("body > section > div > header > input")).sendKeys(Keys.RETURN);

    driver.findElement(By.cssSelector("body > section > label"));
    String text = driver.findElement(By.cssSelector("body > section > label")).getText();

    Assert.assertTrue("Assertion test", text.equals("nightwatch"));

  }

Reporting

This is one area where NodeJS tools have a clear drawback. Since the whole testing is based on response time and efficiency, there is, therefore, no place for sophisticated and rich HTML5 reports. Nodejs tools are limited to the console output and which yields limited feedback. There are a few plugins, which improves that, however, those tools are nowhere near ready to replace Java equivalents.

NightwatchJS simple report
NightwatchJS simple report

Communication Problems

Both Node.js tools introduce an additional step in communication with Selenium-server. However, both of the tools are far from mature. HTTP protocol has no retry, timeout or content length type in GET and DELETE requests. So in case of queuing or parallel running both of the tools are going to break very easily.

All in all, Nodejs integration testing tools are bringing a new shift in the field; moving integration testing from bulky, clumsy and flaky to a more lightweight, efficient and easy to maintain approach. Unfortunately, these tools are still not mature enough to use in a corporate environment, since both of them lack stability and some of the important features. In addition, the shift towards lightweight integration testing still hasn’t caught that much traction so it is not something that everyone is looking for currently. Hopefully, with time, the tools will become more and more popular as Javascript grows in popularity, which will make for a bright future for Nodejs integration testing tools.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s