Creating a Simple Selenium Testing Automation Infrastructure

This is a quick start point guide for anyone who wants to build a simple, efficient and easy to use test automation infrastructure. If you are a QA, who has a good amount of personal automated test cases, a small company or just a department lead who is thinking to start automating testing, this is a good place to start.

What are advantages of building a simple test automation infrastructure?
First of all, it’s availability. With this small testing infrastructure, you can offer automation services for a fair amount of people. I am talking about 20+ constant users who are very active. If you dedicate a person for Jenkins support this number can be increased to 50 or even more. Secondly, as soon as you set up, it requires almost no effort to add and maintain test cases, just copy your Jenkins job, make a few changes here and there and you have a new test case in your CI. And lastly, stability and feedback. Its centralized structure allows you to see and control everything on Jenkins. You don’t need to search for result anywhere, reports, failures, screenshots are all on CI server. All in all, if you have more than 3 people working on test automation or maintenance it would be your interest to try and use something similar.

How do I start? What tools should I use?

First of all, you need to decide what tools will you use. These are the options.

  • Continues integration server – JenkinsHudsonBamboo
  • Selenium WebDriver –  Selenium
  • Physical Machines or VM – You can find it on the internet. Or use one/two physical machines at the start.
  • Source Code Management Repository – Depends what you use, anything is good, i.e BitBucket, Github, etc.
  • Build automation and dependency management system – MavenAntGradle
  • Testing Framework – TestNGJUnit

Steps:

  1. You never want the user to run automation process or test case on their PC. In addition, you should avoid running test cases on your own PC as well. Th process should be isolated and controlled. For that reason continues integration server, like Jenkins, should be used. By using CI you or your user will be able to trigger jobs, see the status, job list, reports and much, much more.
  2. In order to start the test case using Jenkins job, you’ll need to check out your code. Jenkins will need a repository to access and compile the code. This is where Git/GitHub/RTC comes in.
  3. As soon as the build is initiated and compiled by CI (ex. Jenkins) it will need to download all dependencies and set up a build. For that, you’ll need to use Maven & Ant or Gradle.
  4. As soon as you have everything set up your framework will need to start the test case. For that, you’ll need to use one of the testing frameworks like JUnit or TestNG ( if Java is being used). It will kick off your test case, collect information and build a report.
  5. Your test case will be using Selenium WebDriver. Selenium framework should be using PageObject and PageFactory patterns. If you want to use an external data source for your test case, you’ll have to use data provider, which is basically a 2D array.
  6. So your test case is ready to start. Now Jenkins should use one of the available slaves to launch the test. It depends on what are you using, VM or Physical PC.

This is basically a high-level process of your test automation infrastructure and how it is supposed to work. As soon as your test case ends, testing framework will generate an HTML report and provide an answer to the outcome to Jenkins. Jenkins will display a status of the outcome, console output and report (if you set up to get the report). Also, you might be able to access the screenshots of your test case form your slave machine through the Jenkins if you have the right set up.

Building this kind of test automation infrastructure might take days or even weeks, it’s not an overnight process. I could also write a separate blog for each step and tool since each and every one of them requires good understanding in order to configure, however, this is just a high-level model.

In the next post, I’ll dive a little deeper into the 3rd, 4th and most importantly 5th step. I’ll explain how a basic Selenium testing framework should look like.

When To Use Selenium

Lately, Selenium became a go-to tool for almost every QA and SDET. You probably will have a hard time finding a QA/SDET job ad which doesn’t include Selenium in the job description. In one of my previous companies, even manual testers without any programming or automation knowledge and experience were advised to use at least Selenium IDE while developers were pushing them to use their own ‘special’ Selenium testing frameworks.

I have spent some time working as an automation engineer in one of the leading Scandi banks. I was hired to work mainly with JAVA/Selenium, one of my main goals was to help the team to develop and set up a functional, stable (haha) and maintainable Selenium testing framework. In addition, the project owner that I was working for wanted me to automate massive regression tests explicitly using Selenium. So I had no choice, but to put all of my effort and heart into the new testing framework. Luckily I had amazing colleagues who were making great progress on that. So this is my background and how I came to learn Selenium.

I have to say automating huge regression test which has about 30-70 input fields and several different scenarios which varies at runtime was more than challenging. Having a Jenkins job which takes about an hour to run was even worse, especially when it fails so often and the developer of the case has to go through terrible, terrible console output to find hints why the case might have failed and finally connect the points and draw the conclusions. Not to mention constant changes in requirements during the developing process or even post-dev stage.

This for me is a great example when Selenium is overused and overrated. Everyone who used it knows that it’s slow and very unstable, however, it is very easy to fall in the trap of overusing it, since it’s so powerful when it comes to front-end testing.

Test automation is all about the speed and stability, as a QA/SDET, you have to deliver as much information about the state and quality of the product ASAP, to both sides (developer and product owner/stakeholders). If your test lacks speed then obviously the response time suffers greatly, no feedback from devs means that they’ll move on to the other task. This slows down the sprint, makes QA and DEV process more complicated since it’s harder to communicate. The other important factor is stability, if your tests are flaky and unstable, no one will take them seriously and this happens way too often. Unstable autotest equals to no test, you have to avoid that at all costs. So that only means that having good AT’s, Binding tests and UT’s for the developers is much more important than any other information.

What do product owners want from your tests? That’s a good question, most of the time they’ll only need a confirmation that everything works. If you have a non-technical PO (like I did once) it will be harder. They don’t care that much about the speed, however, they care about the quality of the test, stability and most of all a report which looks cool. This is exactly what Selenium does (except stability), it has great reporting tools and there’s nothing better for someone who’s not technical to see how the test executes on the screen or knowing that it’s possible to screenshot everything and basically imitate the user. Product owners (non-technical) will be too excited for all of that and that’s usually the reason why Selenium is sometimes overused and overrated.

Don’t get me wrong I’m not saying that Selenium is a bad tool and you shouldn’t use it. It is a great and powerful tool (especially for performance and integrations tests) but it has it’s used and trying to make it work for AT’s or covering everything is not efficient, stable or fast. I would always use Selenium as the last ditch effort in order to just to make sure that everything is in places. Integration test shouldn’t exceed 30 mins and should have a clear indication of the outcome. If it failed then it should be red and fixed immediately, leaving it red and ignoring the issue is the same thing as not having a test at all.

68747470733a2f2f646f63732e676f6f676c652e636f6d2f64726177696e67732f642f316c4972645a3052526b314a3175596476727247316e4577783752304d6c4f55756745314d6768745630776b2f7075623f773d39363026616d703b683d373230

ReportNG

reportng

Reporting plays a very important part of test automation and ReportNG is a very helpful TestNG plugin which is powerful and easily customizable. Actually, the extent of customization is huge.

This is going to be a quick intro tutorial to ReportNG, my goal is to quickly show how to set it up and add basic functionality.

How to set up?

Download reportNG library

Download Guice

Download Velocity dep

  • reportNG – HTML report plugin for TestNG
  • Guice – Dependency injection framework
  • Velocity dep- Java-based template engine

Next step.

Click on your testNG test case file which you want to run and find “TestNG” and click on “Convert to TestNG

reportng

Now you’ll see the generated XML windows, you’ll have to add these four listeners to the XML file.

So the generator screen will look like this:

Capturelistners

Now when you have generated an XML file, click on XML file and choose TestNG -> Run as TestNG

So this will create a new directory in your project:

test-output -> HTML

Here you’ll find index.html which will have the basic reportNG template.


Improving ReportNG

You can add screenshot and text to the file by using Reporter.log(“””) this is TestNG specific function which adds a line to the report file. Since we are using ReportNG which is HTML5 compatible, we can easily add short HTML “scripts” and it will execute.

Add an Expand/Collapse Screenshot:

Reporter.log(<details> <summary>Click to expand</summary><img alt="" /></details>");

Just change pathToScreenshot to your full path which includes the file name. the expand collapse will mainly work only on chrome.


Advanced stuff

You can achieve much more and add almost anything to the report file by downloading the git source from
ReportNG Git

Most of the stuff is located in ReportNG-base.java, however, you can find useful stuff through all of the projects. You can inject jQuery library and use it to add the script to the template so it’s possible to make these expand/collapse images for all of the browsers, also you can edit almost everything there, adding your logo, descriptions, CSS improvements and jQuerry components.

reportNGexample

Handling tables

Handling tables is something that I saw a few people on the internet and in my company have struggled a couple of times. You can do so many things with tables if it’s done right. So in this short post, I will talk about handling elements within the table.

In one of the previous blogs, there was a snippet how to extract all elements to the list. When you have it on the list you can make all kinds of stuff. For example usually, you would need to check or assert if table element contains some kind of text or if it has changed. When you get the list of full table elements just go through it and assert/check what you need.

		for(WebElement e : list){
			if(e.getText().contains(fileName)){
				return true;
			}
		}

That’s an easy one. But because you add it as a full web element it maintains all of the functions of web element so you can do all kind of stuff with it. Also, because you are in a loop and going through these elements one by one it means that the selected element is isolated and there should be no fear to use some stuff which would be considered dangerous in the plain site, outside the element.

Example:

		for(WebElement e : list){
			if(e.getText().contains(fileName)){
				e.findElement(By.tagName("input")).click();
			}
		}

As you can see I’m not afraid to use tag name input, which in this case mean checkbox because the WebElement is isolated and I’m searching stuff within it.

XPath/CSS int the table

If you are trying to click table link or checkbox and you can’t find anything to catch on don’t be afraid to use regular for loop.

		for(int i = 0 ; i &lt; list.size() ; i++){
			if(list.get(i).getText().contains(fileName)){
			driver.findElement(By.xpath("//td[@id='table:"+i+":subT_rowHeader']")).click();
			Thread.sleep(3000);
			}
		}

This is a good example how can you go through the table elements and click certain things that are hardly reachable, however you’ll need to spend some time finding out the best CSS/XPath pattern for the checkbox/link/expand arrow.

All of these examples work well even if used on a grid with multithreading and the table is manipulated, expanded, customized so take care!

Get all data from tables

This is a simple solution to get all of the data from a table as web element. Later on, when you get the web element you can manipulate it as much as you want, select, compare, assert.

public List extractTableElementsToList(ETable tbl){
    System.out.println("Entering extract all elements from the table via helper.");
    List allTable = null;
    allTable = tbl.findElements(By.tagName("tr"));
    return allTable;
}

This will return the list with all visible table elements.

If you have a case that your table expands and reveals more content after you click on previous elements, you can always get the list of table child elements by using a simple script:

public List extractTableChildElementsToList(ETable tbl){
    System.out.println("Entering extract all elements from the table.");
    List allTable = null;
    allTable = tbl.findElements(By.tagName("td"));
    return allTable;
}

It’s essentially the same but uses TD instead of tr which indicates a child in a table, so this collects all of the visible table child elements and adds it to the list.

Auto download in Firefox

So today I came up with a few challenges, uploading and downloading files from an external server.

Uploading wasn’t that difficult, I have used awt.Robot which handled windows native window pretty well, although it’s very sensitive and I would only advice to use it when you are running test cases on your own dedicated virtual/physical computer.

Downloading was even more tricky. Especially because my framework uses a threaded browser which besides profile requires some capabilities to be implemented.

This is the solution:

firefoxProfile.setPreference("browser.download.folderList",2);
firefoxProfile.setPreference("browser.download.manager.showWhenStarting",false);
firefoxProfile.setPreference("browser.download.dir","\\\\bXXXXX\\workspace\\TEST_DATA\\savedFiles");
firefoxProfile.setPreference("browser.helperApps.neverAsk.saveToDisk","text/csv, text/plain, text/txt, text/html, application/octet-stream");

What does it all mean?
this is explanation line by line:

    • browser.download.folderList tells it not to use default Downloads directory
    • browser.download.manager.showWhenStarting turns of showing download progress
    • browser.download.dir sets the directory for downloads
    • browser.helperApps.neverAsk.saveToDisk tells Firefox to automatically download the files of the certain type

here are a few useful types for the last line:

  • text/html for normal web pages
  • text/plain for plain text
  • application/octet-stream meaning “download this file”
  • application/x-java-applet for Java™ applets
  • application/pdf for Adobe® PDF documents.

Is element visible?

For me and I believe for any decent automation framework is very important to have a reliable method which checks elements visibility before interaction and returns a boolean value instead of exception. Even though it’s an easy solution but for those who just started it will save a lot of time.

this is a neat and very useful method

 

public boolean isElementVisable(By by){
return driver.findElements(by).size() > 0;
}

 

However if you are using PageFactory this gets a bit tricky since WebElement doesn’t have .size() method and getSize returns coordinates which are not useful to make a check. I have tried to make a method avoiding exceptions but in this case it seems that you’ll have to handle it.

 

public boolean isElementVisable(WebElement element){
 try{
   return element.isDisplayed();
 }catch(NoSuchElementException e){
   return false;
 }
}

 

I know that it’s a bit messy solution since exceptions are involved, also webdriver can throw ElementNotVisibleException or something else so this makes it hard to control, unless you throw in a generic exception or catch-all possible ones.