Testing automation has been around for a while but it’s still not always clear how it should work. Just like DevOps, it’s a relatively new role which should improve SDLC process, however, it’s not an easy task to efficiently and effectively implement both roles into traditional software development life cycle. In this blog post, I will share my experience, opinion, and views on how to make the most out of test automation role.
I have been working in this field for the last 5 years. Even though, the experience itself is not that impressive I had a chance to try myself out in a variety of different fields and companies such as an investment bank, smaller software house, information and data company, retail banks. All of these companies were in different sizes, ranging from 50 to 100k+ employees. On top of that, I have attended a large number of extensive interviews most of the time reaching final stages in companies such as Goldman Sachs, Royal Bank of Canada, Barclays Capital, UBS, Bloomberg and some larger hedge funds. And after talking with so many people from so many companies and combining my own personal experience I started to realize that there is no standard of how test automation should fit into SDLC. I have combined all of my different experiences and conversations about the teams and testing process during interviews and I have decided to write down common practices and my views.
Majority of the banks still use waterfall methodology, most of the time it has some agile features like scrum calls, Jira boards, and sprint demos but it’s still essentially a waterfall because the teams in SDLC are separated. In this scenario, BA’s and business pass requirements to developers and they pass implemented features to QA team. Quality assurance, in this case, is a separate team which works independently from developers. In this set up it’s much harder to have effective automation team because automation as DevOps is created to make the process more agile, flexible and quicker, however, boundaries between developers and automation team doesn’t allow it to be used to full potential. But these things happen and to make the best out of this scenario I believe these steps should be taken into account:
- Developers concentrate on unit tests for the features that they implement.
- QA concentrates on BDD / isolated component / integration test.
- BDD test should be based on requirements from BA’s, business and developer unit tests.
- Because this methodology is not very agile and flexible isolated component testing and end to end integration tests carry much more weight and value than usual.
- BDD tests lose a little bit of value in this methodology because quick and stable test results (strengths of BBD) are not important in the waterfall.
- Essentially in this setup automation developer becomes a very technical BA who can check the quality of the product in the very late stages of SDLC.
- Because QA is a separate team communication with the developers is a must, they should always know QA automated tests coverage.
This is I believe where test automation shines the most. Working alongside developers with lots of communication and idea sharing creates a very efficient environment and utilizes automation the most. Having a dedicated person who works on improving unit tests, implementing BDD and integration tests for every feature and all of that being plugged into developers CI cycle is great. Here are some thoughts:
- Unit tests should be reviewed by automation tester.
- Automation tester should be allowed to improve and implement unit tests.
- BDD has a lot of value in SCRUM; quick, responsive and stable test suites can be easily and effectively utilized in CI.
- Good unit and BDD test utilization in CI cycle mean that there is less need for clumsy and flaky end to end, black box integration tests.
- Fewer integration tests mean that there can be more focus on performance tests and continuous integration efficiency.
- QA play a very important role in continuous integration/deployment cycle.
- 3 devs /1 automation developer / 1 DevOps, sounds like a great setup.
What to automate?
The scope of automation is so large that it’s very important to know where to start and what to target. From my own personal experience business quite often wants to see two things targeted by automation:
- Regression automation.
- Finding bugs using automation.
Both of these tasks are pretty much mutually exclusive which makes finding a common ground very tough. You either go for time-saving and target regression automation which usually is a very massive and resource exhausting task or you target new features with lower level tests and try to find bugs.
Luckily there is a solution for these kinds of situations and it’s called a testing pyramid.
Following pyramid approach will maximize return on investment and utilize test automation the most. This is my own personal approach:
- Unit tests – implemented by developers and should include negative tests. (very high ROI)
- Unit integration tests – implemented by developers and should include negative tests. (very high ROI)
- Acceptance tests – BDD type tests which are based on business analyst acceptance requirements. Implemented by automation developer. (high ROI)
- Component based integration tests – BDD can be also used for these kinds of tests, should be implemented by test automation developer. (medium ROI)
- System / UI Selenium / End to End black box – should be implemented by test automation developer. (low ROI)