Continuous Integration is a way of creating and further developing software. What makes it distinctive is that new code developed by programmers is integrated into the main repository branch sometimes as many as a few times a day. Thanks to this, one can have even a few versions of an application built and installed in the test environment on a single day. For a QA team maintaining automated tests, following such frequent changes is a considerable challenge.
My article sets out to present what – in my view – are the most important automated tests’ characteristics allowing one to reach their goals more easily, as well as to explain how this knowledge can be used in practice. All my advice is based on my many years’ experience of work on automated tests in the CI environment using a variety of technologies.
The article’s intended readership is mainly people who create or co-create automated test solutions or those who plan to tackle such tasks.
Usefulness of automated tests in continuous integration
For automated tests to be useful, they must quickly deliver reliable information about the application’s quality in Continuous Integration. One glance at the results should provide the answer to the question of whether there were regressive errors in another version. Between versions, there is no time for analysing the tests which ended with a failure for unknown reasons.
Moreover, every functionality change introduced by programmers means, potentially, a need for readjusting automated tests. Similarly to running and analysing results, it should take as short as possible so that the delivery of information about the quality of another version of the application is not delayed.
Bigger functional changes to the application may result in a need to write more tests to check its new areas, such as sending e-mail or text messages, or printable pdf documents. Automated tests should be flexible enough to ensure that adding new modules is fast and painless. Only if the said conditions are met will one be able to quickly provide information about the quality of each new version of the application.
What it means to automated tests
The above-described requirements come down to three characteristics that automated tests should have: speed, easy maintenance, and stability. These are impacted by a great many factors. Below, I’m going to present those which – in my opinion – are of the biggest importance to the attainment of the goal.
I know from my talks with test engineers working on various projects that situations where these factors aren’t taken into account at all are frequent. This state of things may be caused by insufficient knowledge, technical constraints, or disbelief in tests’ usefulness. It is these observations that inspired me to prepare a presentation and write this article.
Test execution speed
In order to be able to have enough time to run an entire set of automated tests between the consecutive releases of an application, either tests must be very fast or one must find a way to bend the space-time continuum and stretch the time allotted to running tests. Due to the fact that the latter option is far from realistic, l would rather concentrate on the speed of tests.
Speeding up tests means, first and foremost, applying good automation practices. Below, I present a subjective selection of actions which best contribute to increasing the speed of test execution.
Unconditional termination of the test should not be applied
It’s very bad practice to apply commands such as wait or sleep in automated tests thus pausing test execution for a definite time. I can perfectly understand how tempting it may be to solve stability problems in this manner – I used to do the same a long time ago.
However, the solution is short-sighted. First of all, one unnecessarily delays test execution by precious seconds by acting in this way (if the file is downloaded in 5 seconds, another 5 seconds are lost). Secondly, circumstances may occur where even 10 seconds is not enough, and the test, having not found the file, will report an error despite the fact that the file will be downloaded in the 11th second.
Therefore, what must be used instead of the unconditional pause is a dynamic wait for a specific condition, such as an element’s appearance on the page or a file’s presence in the folder.
It’s a good idea to limit user interface testing
As far as automated testing of a web application goes, the most time-consuming thing is page loading in the browser. While the application of the headless technology (browser testing without opening the actual browser window) or going down to the level of application services and complete abandoning UI tests will solve the problem of slower execution, it won’t always be possible. Solutions enabling headless testing, such as PhantomJS, don’t guarantee the same application behaviour as that offered by specific browsers. It happens so because of the use of a different engine for rendering web pages (WebKit). It must be also taken into account that tests don’t support certain technologies, e.g. Flash.
A good solution to the problem is, therefore, going down with tests to the level of API. If the application provides an interface whose use is exactly the same as interaction via the user interface, one can significantly decrease the number of browser tests for the benefit of API tests, examining the business logic and skipping the application part displayed in the browser. The difference in speed may be substantial, but one doesn’t perform a detailed analysis of the correctness of the user interface performance, which – depending on the project – may be the key goal of regression tests. As for tests whose purpose is to evaluate the correctness of controls’ behaviour (e.g. in browsers), one has no choice but to perform tests of the application opened in a full browser.
Abandon testing long paths (e2e)
Automated tests should be used for testing single functionalities, thanks to which one obtains a quick answer to the question of which of the application’s functionalities are performing correctly. Automating long paths, ones going through many parts of the application, results in extending the test time, with the tests becoming less stable as there are more places where something could go wrong. Additionally, if there is a failure in some part of the test, further steps don’t run at all, and one has no information about whether certain parts of the application are working properly. This is why, unless one succeeds in solving these problems in a satisfactory manner, I suggest that end-2-end paths be left to manual testing or be limited to a minimum in the automated testing set.
Parallelisation of running tests
Running tests in parallel is a way for the seemingly impossible task of stretching the time that one has to run them. A simple example shows the extent of the advantage resulting from this strategy.
With 10 tests, each of which takes 3 minutes – on average, running them sequentially will last 30 minutes.
Running tests in parallel – the duration of the longest test, i.e. around 3 minutes.
Parallelisation of running tests, however, isn’t easy and it requires a proper design of the automated test solution, in both technical and logical terms.
In order to be able to make automated tests parallel, the technology one applies to write them must support multithreading. In spite of appearances, not all programming languages, or ready to use automation frameworks, offer this possibility (robot framework, for instance, needs external libraries). Before an automation technology is chosen, it’s worth checking if there won’t be problems in this area when test parallelisation is initiated.
A second requirement is the need to have a laboratory with a certain number of machines to run the tests. I can say, from my experience, that 5-10 browsers with tests can be safely opened on a single computer or virtual machine. The number, however, will vary depending on the amount of memory needed to run the application itself in a browser. It may turn out that with a few dozen tests one will need not one but a few computers or virtual machines. Alternatively, one may also use services making virtual machines available, with any browser on request. SauceLabs, Gridlastic, or Browserstack are good examples here, and there are many other suppliers on the market, offering similar solutions.
I’m going to present the way of solving the above-described problems using the example of automated tests which I programmed for the web application fireTMS.com for carrier and freight forwarding companies. The automated tests were written in Java using JUnit, Maven, WebDriver, and SauceLabs/Gridlastic.
Java itself supports multithreading, and thanks to the application of Maven and the Surefire plugin, one is able to comprehensively control test parallelisation from a single place. The configuration looks as follows:
The parameters have the following functions:
useUnlimitedThreads – makes it possible to indicate whether the maximum number of threads is to be used,
forkCount – determines how many test classes are to run simultaneously,
threadCount – determines how many test methods are to run simultaneously,
parallel – parallelisation strategy (other available ones include classes, both, and suites),
The Surefire plugin offers many more settings enabling the management of test parallelisation. These include advanced options such as making the number of threads dependent on the number of cores of the processor on which the tests are intended to run. For complete documentation, please refer here.
I used the Sauce Labs service, alternately with GridLastic, to run the tests. The diagram provided below presents how this kind of solution works:
Commands from the automated script are sent to the Selenium Grid’s hub on the service provider’s server. Next, depending on the test configuration, virtual machines (to which specific instructions of the tests are sent) running required browsers are launched. The web application runs in the browsers, in a given environment, and the tests are carried out in it. If need be, communication between the virtual machines and the application can be secured with a VPN tunnel.
The method provides enormous possibilities of test parallelisation. However, the cost is a drawback. Depending on the plan and number of virtual machines available, it may amount to 20 to a few hundred USD per month.
Independence of tests
Another important matter is the need to design tests in a manner enabling each of them to run on its own.
In every company I have worked for so far, at a certain stage of work there was an idea where – if we already have an automated test for creating an object in the application – we could perhaps use it also for the test of object data edition and data deletion test. In reality, however, it isn’t a good solution – apart from the fact that it completely prevents running tests in parallel, it also results in test results becoming obscured. Let’s see why. An example sequence will look like this:
TC1 – creating the user
TC2 – editing TC1 user data
TC3 – deleting TC1 user
In the event of an attempt to run the elements in parallel, TC2 and TC3 will have a negative result (they won’t find the user) because the user they will try to work on will be created only after TC1 has been successful.
What’s worse, even if the three tests run sequentially, in the event of a TC1 error, the subsequent tests will also end up in a failure or won’t run at all. In consequence, one error and two tests with unknown results will be reported.
If the independence of the tests is ensured, they will look like this:
TC1 – creating the user
TC2 – creating the user (by API), editing user data,
TC3 – creating (by API) and deleting the user.
TC2 and TC3 prepare test data for themselves and on their own, thanks to which one can run these tests in parallel thus saving a lot of time. Moreover, if users for TC2 and TC3 are created without the user interface (e.g. using API), there is a guarantee that if an error occurs in TC1, the other tests will be executed and will return information about whether the functionality editing and deleting user data is working.
The creation of data for every test is ensured with the JUnit annotation @Before, where one invokes API’s that create indispensable data one by one:
- creation of a new tenant by API,
- creation of a tenant division,
- creation of a user for a given tenant,
- creation of the other, indispensable, tenant settings for all tests.
If the test requires additional data, it must be created already in the test method itself.
Thanks to putting the above-described good practices into practice and parallelising test execution, it’s possible to carry out the entire set comprising 240 tests in 30-40 minutes, as against 12 hours that it would take them to complete if they ran sequentially.
Ease of maintaining and adding new tests
The appropriate application of the Page Object Pattern model provides an opportunity for quickly maintaining tests and easily adding new ones. Also, not to be overestimated is the possibility of quickly including new engineers in the project.
Having said that, there are elements which, when applied with Page Object Pattern, prove essential to whether the framework turns out to be good enough in the face of bigger challenges related to UI testing, such as comprehensive changes to the application layout or problems with the infamous Stale Element Reference Exception.
Wrapping WebDriver methods
The interaction between the tests and a browser should be wrapped using the tester’s own methods, thanks to which it will be possible for actions such as click(), getText(), getAttribute(), sendText(), etc. to be extended by adding a wait for the right state of an element or support of animations for application page loading and a progress bar. Proper application of a dynamic wait for elements will also solve the Stale Element Reference Exception problem.
First, the waitForLoader method is invoked and waits until the application loading progress bar disappears. Next, waitForElementClickable should be invoked and wait until the element is ready to be clicked on. In the subsequent line, a WebDriverWait class dynamic wait object should be created, with timeout set at 20 seconds. The subsequent lines serve the purpose of setting another attempt to execute the action at 200 milliseconds and making sure that in case of an InvalidElementStateException and StaleElementReferenceException, they will be ignored and that there will be another attempt to execute the action in the @Override section until it’s successful or until the timeout elapses.
The other example refers to the waitForElementClickable wrapper, which not only evaluates whether an object is clickable with one of ExpectedConditions but also checks, in the @Override annotation, the element attributes characteristic for the application being tested. When I used this method, despite the fact that the ExpectedConditions.elementToBeClickable condition was met, some controls were set as inactive in a way transparent for the ExpectedConditions class. Only after additional conditions were included was it possible to fully solve the problem with WebDriver’s too early clicking on the button.
The section below refers only to cases where, due to the application’s limitations, programmers aren’t capable of placing html node tags in a location of their choice.
Page Object Pattern assumes the storage of locators for controls on the page in a single location, but it’s worthwhile to take a step further. Owing to the fact that web applications are developed using frameworks, our locators are highly likely to have common parts. In such an event, these common parts should be relocated somewhere else, e.g. to an abstract page class, so that it’s easier to manage them. In the application discussed here, all tables are embedded in the application in the following manner:
As it can be seen, the path xpath – “/div/div/table/tbody” is between the node with a unique id and the table node. The path can be stored in a variable in the abstract page class.
Next, it can be used on all pages when creating locators:
Thanks to this solution, a change to the framework or page layout allows quick correction of paths between a node with a tag and a table node.
If it’s possible, tests should have separated methods that may be applied to standard objects in the application, regardless of on which page they are located. Thanks to this, adding another page to the project in POP will take less time, and the engineer coding it won’t have to solve problems which have already been solved.
Below is an example of a method from an abstract page class. It receives a locator in tables, the value to be checked, and the number of line and column where one wants to find it.
Based on my own experience, I can confirm that by applying wrappers, shared locators, and abstract methods one is able to adjust over 200 automated tests after a complete change of layout in three days.
Stability and flexibility:
What I understand by “stable tests” is tests that don’t end up in a failure for unknown reasons. As for flexibility, I use this term to denominate a possibility of running tests in any environment and with any settings, and without a need for changing the very code of the tests, as well as a possibility of quickly expanding tests by adding new modules.
In order to ensure stability, both for open source and commercial solutions, one needs to use a well-tested tool with good access to support. If one comes across a problem involving test instability which they can’t solve on their own, it may mean a situation really difficult to get out of. How popular is the technology to be applied will be quite an important factor in this context. The more people use a method, the bigger a chance that the solution to the problem will be found on pages such as Stack overflow. The risk of unexpected behaviour of the tests is reduced in this way.
A second stability factor is the already mentioned use of wrappers and central management of the browser interaction. Applying these solutions will allow one to completely remove test stability problems stemming from WebDriver’s characteristics such as the lack of support of asynchronous technologies (e.g. AJAX) as well as rid of the trouble related to the wait for elements in a specific condition.
Java and Maven make it possible to meet the flexibility criterion thanks to the parameterisation option and the possibility of expanding tests by adding new modules.
Thanks to the proper construction of an abstract test class and the POM.xml file, one is able to control input parameters for the tests exactly where they are to run and determine in what environment and in how many threads they will run simultaneously.
Applying Maven to run tests enables sending a group of parameters to them:
Using variables in the abstract test class:
A WebDriver’s object is appointed with the choice of the server to which its requests are to be sent. The server is determined in accordance with the parameter sent by Maven:
The getInitialUrl method collects the application URL from the parameter sent by Maven. The method is later used as an argument of the login method in the application:
Passing variables on to POM.xml is carried out in the following way:
Thanks to parameterisation of the main test class and the POM.xml file, one is able to change all significant test environment parameters without interfering with the code, which considerably speeds up and facilitates switching between test environments.
Possibility of expanding tests by adding new modules
Let’s imagine that the application obtains a module that enables sending e-mail messages or printing pdf files. In order to ensure automated testing of these areas, one must have a possibility of easily importing new libraries to the automated test project. Some open source frameworks have a limited number of libraries created by the community with limited support and slow reactions to changes. Neither do commercially available solutions support all technologies. The intention of using additional modules often means a need to purchase an extra licence – an issue which one should bear in mind when choosing solutions for automated tests.
In the event of applying Java, the action to be taken comes down to finding a library which suits the set goal (usually there are a few of these), making sure it’s located in Maven’s central repository, and adding an entry to POM.xml, e.g.:
After rebuilding the project, one has access to all library classes.
Speed, confidence in results, and a possibility of easy expansion are the features that automated tests should offer to be considered useful in Continuous Integration. Daily work on a few versions of an application causes automated tests that don’t meet these parameters to be unable to quickly provide information about the presence of regression errors in the application.