WebTest vs Selenium: WebTest wins 13 – 5

In the last months I’ve seen a rising interest in automated testing of web applications thanks to the efficient viral marketing of Selenium. However, the world is full of test automation projects that started with big hopes and lots of enthusiasm only to be abandoned shortly after facing the unpleasant reality that it needs more than a point-and-click activity to develop a suite of robust tests.

The maintainability of automated tests depends primarily on the skills of the test authors, but different tools have different features that impact their efficiency. This blog post compares the features of two open source automated web testing tools: Canoo WebTest and Selenium.

A short introduction to the contenders:

Canoo WebTest is a free open source tool that has existed since 2001. It is written in pure Java and contains a set of Ant tasks that drive a simulated, faceless browser (originally HttpUnit, but for the last few years HtmlUnit).

Selenium is a free open source tool as well, created in 2004. It uses injected JavaScript to work within real browsers. Different components exists under the name Selenium: Selenium Core, Selenium RC, Selenium IDE (!), … In this blog post I will only consider Selenium RC used with Selenium IDE on Firefox or Selenium HTA on Internet Explorer due to the limitations of the other possibilities.

Features comparison

To be clear, as a WebTest (and HtmlUnit) committer I’m undoubtedly biased. On the other hand, I have experience with huge functional test suites being developed and maintained over periods of years. Trying to be objective, I may overcompensate in the other direction and give Selenium too much credit. Of course I will diligently fix errors I may have in my Selenium understanding. But please read this post until the end before starting with criticisms ;-)

  • Browser fidelity: WebTest 01 Selenium

This is probably the most overestimated characteristic of a web testing tool. Automated tests don’t make manual testing useless because automated tests can’t cover everything (at least for affordable costs). You still have to walk through your application (just think of everything you’ve checked just reading this: page layout, font size, font colors, …).
The consequence is that an automated web testing tool’s purpose is not to ensure that an application works “well” as it is not possible, but to detect most of the failures that could happen. This is a huge difference because it means that tests don’t have necessary to run in a “real” browser.
Nevertheless the browser’s real behaviour has to be approximated as closely as possible. HtmlUnit’s JavaScript support has made impressive progress but it still doesn’t (and will never) behave exactly like a normal browser.
Even though Selenium modifies the normal JavaScript execution of an application, it uses the browser itself and therefore is nearer to the standard behaviour of the browser.

  • Reports: WebTest 10 Selenium

JUnit-like reports are far too limited for web test automation. This is probably something that you first see when you have reached a certain volume of tests. If the tests are successful, you don’t need any report at all, but when some tests fail, you need the information to find as quickly as possible what is the failure cause and an error message is often not enough.
With comprehensive reports like those provided by WebTest, you don’t have to debug your tests, just to analyse the reports. Furthermore it allows you to understand (and fix) the worst kind of bugs: those that don’t occur systematically.

  • Speed: WebTest 10 Selenium

Tests are never fast enough. Selenium is known not to be very fast and even slower on Internet Explorer (just read the mailing list) and seems to suffer from memory leaks. On the other side, WebTest is quite fast (see for instance this thread in Selenium Dev mailing list for a non representative test where Selenium took ~ 10 seconds and WebTest < 2s).
It’s not surprising due to Selenium’s architecture (3 tiers involved) and all the rendering that happens in browser. Even if HtmlUnit’s HTML handling algorithms are not as good as the real browsers, WebTest has simply less to do and everything happens in the JVM.

  • Integration in development process: WebTest 10 Selenium

WebTest is “just” Ant which means that it can directly be called from CruiseControl for instance or from each developer’s workstation.
On the other side for Selenium you need a real browser with an own profile, a proxy – possibly on another computer if you want to test with IE and run the tests from a non Windows system.

  • Scalability: WebTest 10 Selenium

For a large application (or set of applications) with a good functional test coverage your test suite(s) will rapidly grow and scalability may become an issue. WebTest scales far better than Selenium mostly because it’s faster and because you can simply run many tests suites in parallel (just think of the hardware requirements and browser limitations to do that with Selenium).

  • Capture JS errors: WebTest 10 Selenium

This is what surprises me most about experienced developers working with Selenium: they find it acceptable to ignore JS errors. Would you accept compilation errors in your program as long as your unit test pass? Surely not! But in fact this is exactly what you do with Selenium as it doesn’t detect the javascript errors contained in your application (unless they directly impact the specific tests causing them to fail).

  • Testing AJAX: WebTest 11 Selenium

Contrary to popular belief, you don’t need to run your test as JavaScript inside a browser to test AJAX functionality. HtmlUnit and thus WebTest is just as well up to the task. It can even be considered superior as it allows better control over how to schedule the in-page requests making the unpredictable browser behavior predictable (see for instance my previous post).

  • Beginner friendly: WebTest 01 Selenium

Beginners (as well as managers ;-)) better understand test automation of web applications, when they see what happens.

  • Documentation: WebTest 10 Selenium

Extensive and up-to-date documentation is very important. A quick look at both web sites will show you that WebTest manual is clearly the winner. It should even be a negative point for Selenium as advice on what makes test suites maintainable is completely missing.

  • Predictable behaviour: WebTest 10 Selenium

Should be a minimal requirement for a test tool, but if you look regularly at Selenium mailing lists or at different posts (like this one), this is not yet fully the case for Selenium.

  • XPath support: WebTest 10 Selenium

WebTest currently uses Jaxen as XPath engine which means that XPath 1.0 is covered as well as some XPath 2 functions (do you know that starts-with is in XPath 1 spec but ends-with first appears in XPath 2?).
Additionally you can customize it to define your own XPath functions.
Selenium uses native XPath support when it’s available (like in Firefox) and evaluates XPath expressions using JavaScript libraries otherwise (like in IE). This JS library is slow and many XPath expressions aren’t interpreted correctly. Even in Firefox, the support is limited to XPath 1.0.

  • Support for badly formatted HTML code: WebTest 01 Selenium

Browsers are able to cope with really badly formed HTML code and so does Selenium as a consequence. WebTest’s parser (NekoHTML) is able to handle some malformations but not that much. Even though it is quite questionable to see it as a feature when your goal is to write your web application as well as possible, sometimes testers do not have access to the development resource to correct the source and just want to test functionality so I’ll give this point to Selenium.

  • Extensibility: WebTest 10 Selenium

Selenium accepts custom extensions but first this is cumbersome because the extensions have to be deployed in the target browser(s) and second interactions are limited as extension code executes in the browser and not in your test program.
In WebTest you have full control over the “browser” from within your tests which you can use to simply write global extensions as well as project or test specific ones.

  • Data driven tests: WebTest 10 Selenium

No discussion, the dataDriven Ant task used with WebTest is simple and powerful!

  • Multi-language support: WebTest 01 Selenium

Selenium RC has bindings in different languages (Java, Ruby, PHP, …) whereas WebTest is bound to Ant which means XML or for instance Groovy with its nice AntBuilder (in fact any of the over 200 languages for the JVM could probably be used). I think that Selenium is missing a real specification language (please don’t talk about Selenese!) like Ant is in this case for WebTest, but I need to give some points to Selenium…

  • Internationalisation support: WebTest 10 Selenium

Using WebTest, you just need to put your language specific strings in property files and use Ant’s built in property task to load the right resources before executing your tests.

Update 05.11.07:
Dan Fabulich correctly indicates that Selenium RC tests written in the same language than the AUT can directly use the application’s i18n resource bundles. Therefore WebTest can’t be seen as winner outside of the Java world.

  • Support for non HTML content: WebTest 10 Selenium

HTML is only one of the content types used by a web application and it’s a common need to have mixed content within the same applications with for instance a PDF file containing the invoice after the checkout. Selenium is limited to HTML content (+XML and text). On the other side WebTest has built in support to work with PDF documents and Excel files as well as Applets and Emails.

Conclusion:

Automated functional tests of web application should become as natural as unit tests. Some tests are better than no tests, no matter which tool is used. Selenium does a good job to introduce newcomers and has many advantages (besides the price) over its commercial model QTP. Nevertheless at least when the size of your suite grows, you should pay attention to your efficiency if you want to last and “the ROI on WebTest is many orders of magnitude higher than any other tool I’ve used” (Lisa Crispin, author of Testing Extreme Programming).

When comparing Selenium and WebTest, 3 categories of web application can be considered. First the applications that are supported both by WebTest and Selenium. This contains most of the applications. Then the ones that use browser features (mostly javascript) that are not yet supported by HtmlUnit. Due to HtmlUnit’s awesome progress in JavaScript support, the size of this category continuously shrinks. The last category concerns applications that use for instance PDF documents or applets and for which Selenium has no support. In all cases where WebTest can be used it is far more efficient than Selenium to ensure the quality of web applications because it provides more feedback and takes less time (both to execute and to analyse results).

When I started writing this post I didn’t expect that Selenium would get such a bad score. Comments are welcome to show which advantages of Selenium over WebTest I’ve missed!

HtmlUnit vs HttpUnit

HtmlUnit committer Daniel Gredler has written an interesting comparison of HtmlUnit and HttpUnit in his blog. I particularly like the analogy with Struts:

“HttpUnit is to the web app testing world what Struts is to the web app framework world: there are many “better” options out there, but it just won’t go away!”

Daniel Gredler blog post: HtmlUnit vs HttpUnit.

Thoughts on Selenium Grid

I’m impressed by the activity around Selenium. Not because I’m impressed by Selenium but because I think that all this work could be better invested. For instance it would surely be a smaller effort to make for instance HtmlUnit able to work with its recognized speed with numerous browser configurations, simulating perfectly their JavaScript execution, and not only 2 browsers as Selenium really supports. The last Selenium related project I’ve discovered is Selenium Grid.

Run tests in parallel

Selenium Grid’s aims to allow running tests on multiple computer because:

  • Selenium remote control is quite slow at driving the browser“.
  • You can only run a limited number of concurrent tests on the same remote control before seriously impacting its stability. Practically speaking, launching more than 6 browsers on the same Selenium Remote Control is not advisable. The limitations are even more drastic for Internet Explorer.

In other words: slow and not scalable. If I correctly understand the documentation, Selenium Grid will allow to run approximatively 6 x N tests in parallel, where N is the number of computers in the grid. Not really impressive.

An HtmlUnit user reported that he performed load testing with HtmUnit using 350 threads in the same VM and one WebClient instance per thread. For this purpose he allocated 512M of his 1Gig laptop to the VM and it worked very well. WebTest surely adds some overhead to HtmlUnit “pure” but, due to the indications of this user, I could imagine that it would be possible to run 500 threads of WebTest tests on one computer within a single VM. This would save just… 80 computers compared to Selenium Grid!

Parallelisation often limited

Selenium Grid’s documentation says: “Selenium Grid cuts down on the time required to run a Selenium test suite to a fraction of the time that a single instance of Selenium instance would take to run“. Sadly this isn’t that easy. Not all tests can be run in parallel: as soon as you test the functionalities of registered users, you’re often limited to a serial test execution (or a just a few parallel executions) when your application doesn’t accept numerous simultaneous login of the same user and you don’t have so much users available with the right settings for the tests. In this case only pure speed can help.

Grid computing is an exciting large area, perhaps is Selenium Grid only a side product of some more significant activity…

script.aculo.us drag & drop now works for HtmlUnit

I wanted since a long time to be able to test drag & drop features with HtmlUnit like those of script.aculo.us demo shop.

This was a long work to implement all missing js functionalities. Particularly script.aculo.us’ drag & drop requires that information on elements’ position in the page are available but HtmlUnit doesn’t compute any rendering information (what makes it very fast). The solution was to compute faked positioning information: the position isn’t important by itself, it just needs to be coherent with the information of the mouse event.

The result is that it is now possible to test drag & drop on the demo shop:

public void testDragAndDrop() throws Exception
{
final String url = "http://demo.script.aculo.us/shop";
final WebClient webClient = new WebClient();
webClient.setAjaxController(new NicelyResynchronizingAjaxController());


final HtmlPage page = (HtmlPage) webClient.getPage(url);


final HtmlElement mug = page.getHtmlElementById("product_1");
final HtmlElement cart = page.getHtmlElementById("cart");
assertEquals("check that the basket is empty", 0,
cart.getByXPath(".//img").size());


mug.mouseDown();
cart.mouseMove();
cart.mouseUp();


assertEquals("check that the basket now contains 1 element", 1,
cart.getByXPath(".//img").size());
}

Quite simple, isn’t it?

Update 31.03.08:

I’ve changed the XPath expressions from //img to .//img for compatibility with upcoming HtmlUnit 2.0.

XPath Power (2): Detect Lack of Experience

A lot of people feel qualified to talk about test automation of web applications, probably because the principles are really easy: tests have “only” to do what a “normal” user would do. In the practice many of these persons don’t have real test automation experience and you shouldn’t trust their statement directly.

The problem is that it’s difficult to determine if a person has a real experience and therefore if the proposed solutions have proved their worth or not. Declarations from a person that hasn’t written tests receiving at least a few hundreds of responses from the web server are questionable. Exactly the same is valid for people talking from recording tools without to give any advice on what can be expected from the recorded scripts.

An other way to detect lack of experience is to look at the XPath expressions used. XPath support is indispensable in test automation of web application but depending how the xpath expressions are written, they may be far too “strong” and fail on the first unrelated change in the tested html page.

A few “bad” XPath examples
Here are a few examples of “bad” xpath available in blogs, conference slides or even tool documentation:

/html/body/table[2]/tbody/tr/td[2]/table[2]/tbody/tr/td/div/ul[1]/li[4]
from http://agiletesting.blogspot.com/2006/01/useful-tools-for-writing-selenium.html

//body[1]/table[1]/tbody[1]/tr[1]/td[1]/img[1]
from http://www.artoftest.com/Resources/WebAii/Documentation/topicsindex.aspx?topic=elementid

//html[1]/body[1]/div[1]/div[1]/h3[1]
from http://artoftestinc.blogspot.com/2007/01/ajax-testing-and-waiting-for-page.html

//html/body/div[@id='outerDiv']/div[@id='innerDiv']/div
[@id='pinDialog']/table/tbody/tr/td

from http://www.nealford.com/downloads/conferences/canonical/Neal_Ford-Advanced_Selenium-handouts.pdf

/html/body/table/tr[2]/td/table/tr[1]/td/a
from http://wiki.rubyonrails.org/rails/pages/HowtoFunctionalTest

XPath Power

XPath support is indispensable for any serious web test automation tool because xpath expressions allow to precisely select nearly everything in an HTML page using a concise syntax.

Last week I had again a case showing the power of XPath. Colleagues had to write a WebTest script that should (among other steps) change the value of a select like this one:

<select name="mySelect">
<option value=""></option>
<option value="1">English</option>
<option value="2">French</option>
<option value="3">German</option>
</select>

(this blog doesn’t allow to “display” this select field it as it would normally appear, but you have surely enough HTML knowledge to understand it)

No assumption could be made on the current value of the select, the test had just to select an other valid option (meaning not the first one).

First approach
The first approach could be to read the index of the selected option, increment it, perform a modulo operation and avoid the first option. More pragmatic would be to take the second one (here “English”) if it is not already selected and the 3rd one (here “French”) if “English” was selected.
These solutions work but require some logic that can only be expressed in a script.

More elegant solution with xpath
In fact you can simply formulate the test specification with an XPath like this one:

//select[@name='mySelect']/option[text() and . != ../option[@selected]]

This XPath just selects all the options of select field mySelect that have a “content” (i.e. not the first, empty one) and that are not currently selected.

——
NB: WebTest’s setSelectField step takes automatically the first option when the xpath evaluation returns more than one result. For other test tools, you may need to select explicitly the first option like this (//select[@name='mySelect']/option[text() and . != ../option[@selected]])[1]

HtmlUnit: Re-synchronize AJAX Calls for Simple, Deterministic Test Automation

The upcoming release of HtmlUnit will contain a very large number of fixes and improvements (particularly for the JavaScript support of libraries like Prototype or GWT). An interesting new feature is the experimental AjaxManager that allows to easily control the execution of AJAX calls.

As a user you often don’t even notice that the changes performed in an html page have been performed asynchronously. But this is just because you’re too slow and automated tests (which are very fast) will encounter a problem: they will want to perform some action on the page before this one has been updated.

Current approaches
Automated tests tools propose currently 2 approaches (*) to face this problem and (try to) ensure a deterministic test execution.

  • add an instruction to sleep for a certain amount of time before executing the next instruction (that’s the case for instance of tools like QTP or Squish)
  • add a waitXxxx instruction to wait until some element is available in the page (like for instance in WebAii or Selenium), element that gets created by the AJAX call

The first strategy is very rudimentary: if you don’t wait long enough, you may see your test failing just because the web server is more busy than usual. And if you wait too long, you will end up with test suites taking hours to finish. The second strategy has not this problem but requires the specification of an additional condition (that has to be maintained when the application evolves).

Drawback
In fact none of these strategies has tested that some call has been performed asynchronously (and this test was not required) and both require additional instructions that don’t really belong to the functional test.

Resynchronize (some) AJAX calls to allow simple, deterministic testing
Upcoming HtmlUnit release will allow to control AJAX execution and to turn asynchronous calls back to a synchronous execution mode. The consequence is that the next instruction of the test program is first executed when the AJAX execution is completed. This makes usage of sleep or waitXxx-like instructions needless what simplifies the test conception.

Nevertheless not all AJAX calls need to be turned synchronous. Typically calls that are not triggered by a user interaction like those from Prototype‘s Ajax.PeriodicalUpdater don’t be need to be re-synchronized (**). For this purpose HtmlUnit now contains the experimental NicelyResynchronizingAjaxController class which can be used to resynchronize only the AJAX calls made from the main execution thread which are normally the ones triggered by user actions.

Limitations
This feature is very new in HtmlUnit and the experience is missing to make a definitive evaluation. The re-synchronisation worked just fine for the application I had to test and allowed me to eliminate many lines of code. Nevertheless there are still cases where it wouldn’t probably be enough. If I correctly remember, when Prototype‘s Ajax.Updater for instance is configured to evaluate the scripts it receives, it uses a setTimeout to perform this task after having treated the rest of the response content. The consequence is that the action of these scripts wouldn’t be intercepted to be executed in the main “flow”. At least in this case the time consuming server round-trip would have been catch, allowing a far smaller “sleep” delay.

Could this technique be used by other test tools than HtmlUnit?

WebTest will of course benefit soon from this technique but this would probably make sense for other web app testing tools as well. This wouldn’t be a problem for those who simulate a browser as they are exactly in the same situation as HtmlUnit and have plain control over AJAX execution. For the tools that work within the browser using JavaScript, I guess that it could be possible to redefine the open method on the XMLHttpRequest objects (as well as on MS’s XMLHttpRequest ActiveX variants) to change the value of the asynchronous parameter. It may become a bit more tricky to determine what is triggered by user interaction to perform a selective resynchronisation. Finally for the tools that drive browsers using their API it will depend of the features offered by this API but this is a domain that I don’t know.

——
(*) at least this is what I could find in the documentation of the numerous tools I’ve looked at to write this post (it’s amazing how many – useless? – web testing tools you can find on the net!)
(**) in fact it probably doesn’t matter if resynchronized or not as the execution already runs in a separate thread

WebTest’s New Motto: The Most Effective Way To Test Your Web App!

We’ve change WebTest‘s motto (previously it was “Free open source tool for automated testing of web applications”) to better express what WebTest’s strengths are.

A lot of tools for automated testing of web app are available and have different features that may appear appealing in a first time. WebTest shines in what I call “professional test automation” because it allows to simply write test scripts that are reliable, fast, easy to run, cheap to maintain, helpful to quickly find errors and it scales fine and is easy to extend. Lisa Crispin, author of Testing Extreme Programming, described this perfectly:the ROI on WebTest is many orders of magnitude higher than any other tool I’ve used”.

This new motto is only the first step. Keep tuned!

iX Bessere Software! Conference 2007: WebTest Tutorial & Session accepted

I will present a full day tutorial as well as a session about efficient test automation of web applications in general and WebTest in particular together with Dierk König at the next iX Bessere Software! conference, 19-21 November, in Düsseldorf (Germany).

We’ve already presented a WebTest tutorial at the previous edition of this conference last year in Frankfurt. This was really nice and I think that our tutorial was the most visited one. I hope that this year’s edition will be as good as the last one.

Only Selenium can test it? What? Surely not this simple page

When you look at a conference program, the session abstracts are naturally a kind of publicity to motivate people to attend and should be considered as such. Nervertheless an abstract shouldn’t exagerate too much to stay credible.

“… including applications that only Selenium can test”. Where?

In Neal Ford’s numerous sessions about Selenium, you will see again and again that he will present “testing Ajax applications (including applications that only Selenium can test)” (see for instance in
Advanced Selenium (No Fluff Just Stuff)
). When you write such a provocative statement, you have to prove it. I’ve attended one of his presentations (it was really poor, see previous post here) but I haven’t seen anything that would cause trouble to all existing web testing tools except to Selenium.

But very simple pages can’t be tested with Selenium

On the other side a page as simple as this example

<html>
<head>
<script>
document.getElementById = function(_id)
{
return {id: _id, nodeType: 3, data: 'he he, you failed!'};
}
</script>
</head>
<body id='bodyId'>
this is a very simple page
</body>
</html>

will currently cause enorm problems to Selenium that won’t even be able to execute correctly a simple
verifyText id=bodyId this is a very simple page

The reason? Selenium uses the DOM functions to work on the html document and therefore relies on them.

Ok, this point isn’t a very big limitation as changing the functionality of the DOM functions is not a common need. In fact this post was just to say that I wait to see a web application that “only Selenium can test” but that even a very small page can cause problems to Selenium whereas it is handled correctly by most of the other web testing tools.

« Older entries Newer entries »

Follow

Get every new post delivered to your Inbox.