WebTest Monitor

WebTest now contains a minimal GUI showing the progression of the tests execution when started manually:

webtestmonitor.png

What does it bring? In fact nearly nothing.

It’s just a small information displayed during test execution. It becomes funny when you start a few dozen tests in parallel because you nearly don’t have time to follow each single test that get started or finished: it goes too fast. But this will be the subject of a future post once this feature is integrated in WebTest builds.

WebTest vs Selenium: WebTest wins 13 – 5

In the last months I’ve seen a rising interest in automated testing of web applications thanks to the efficient viral marketing of Selenium. However, the world is full of test automation projects that started with big hopes and lots of enthusiasm only to be abandoned shortly after facing the unpleasant reality that it needs more than a point-and-click activity to develop a suite of robust tests.

The maintainability of automated tests depends primarily on the skills of the test authors, but different tools have different features that impact their efficiency. This blog post compares the features of two open source automated web testing tools: Canoo WebTest and Selenium.

A short introduction to the contenders:

Canoo WebTest is a free open source tool that has existed since 2001. It is written in pure Java and contains a set of Ant tasks that drive a simulated, faceless browser (originally HttpUnit, but for the last few years HtmlUnit).

Selenium is a free open source tool as well, created in 2004. It uses injected JavaScript to work within real browsers. Different components exists under the name Selenium: Selenium Core, Selenium RC, Selenium IDE (!), … In this blog post I will only consider Selenium RC used with Selenium IDE on Firefox or Selenium HTA on Internet Explorer due to the limitations of the other possibilities.

Features comparison

To be clear, as a WebTest (and HtmlUnit) committer I’m undoubtedly biased. On the other hand, I have experience with huge functional test suites being developed and maintained over periods of years. Trying to be objective, I may overcompensate in the other direction and give Selenium too much credit. Of course I will diligently fix errors I may have in my Selenium understanding. But please read this post until the end before starting with criticisms 😉

  • Browser fidelity: WebTest 01 Selenium

This is probably the most overestimated characteristic of a web testing tool. Automated tests don’t make manual testing useless because automated tests can’t cover everything (at least for affordable costs). You still have to walk through your application (just think of everything you’ve checked just reading this: page layout, font size, font colors, …).
The consequence is that an automated web testing tool’s purpose is not to ensure that an application works “well” as it is not possible, but to detect most of the failures that could happen. This is a huge difference because it means that tests don’t have necessary to run in a “real” browser.
Nevertheless the browser’s real behaviour has to be approximated as closely as possible. HtmlUnit’s JavaScript support has made impressive progress but it still doesn’t (and will never) behave exactly like a normal browser.
Even though Selenium modifies the normal JavaScript execution of an application, it uses the browser itself and therefore is nearer to the standard behaviour of the browser.

  • Reports: WebTest 10 Selenium

JUnit-like reports are far too limited for web test automation. This is probably something that you first see when you have reached a certain volume of tests. If the tests are successful, you don’t need any report at all, but when some tests fail, you need the information to find as quickly as possible what is the failure cause and an error message is often not enough.
With comprehensive reports like those provided by WebTest, you don’t have to debug your tests, just to analyse the reports. Furthermore it allows you to understand (and fix) the worst kind of bugs: those that don’t occur systematically.

  • Speed: WebTest 10 Selenium

Tests are never fast enough. Selenium is known not to be very fast and even slower on Internet Explorer (just read the mailing list) and seems to suffer from memory leaks. On the other side, WebTest is quite fast (see for instance this thread in Selenium Dev mailing list for a non representative test where Selenium took ~ 10 seconds and WebTest < 2s).
It’s not surprising due to Selenium’s architecture (3 tiers involved) and all the rendering that happens in browser. Even if HtmlUnit’s HTML handling algorithms are not as good as the real browsers, WebTest has simply less to do and everything happens in the JVM.

  • Integration in development process: WebTest 10 Selenium

WebTest is “just” Ant which means that it can directly be called from CruiseControl for instance or from each developer’s workstation.
On the other side for Selenium you need a real browser with an own profile, a proxy – possibly on another computer if you want to test with IE and run the tests from a non Windows system.

  • Scalability: WebTest 10 Selenium

For a large application (or set of applications) with a good functional test coverage your test suite(s) will rapidly grow and scalability may become an issue. WebTest scales far better than Selenium mostly because it’s faster and because you can simply run many tests suites in parallel (just think of the hardware requirements and browser limitations to do that with Selenium).

  • Capture JS errors: WebTest 10 Selenium

This is what surprises me most about experienced developers working with Selenium: they find it acceptable to ignore JS errors. Would you accept compilation errors in your program as long as your unit test pass? Surely not! But in fact this is exactly what you do with Selenium as it doesn’t detect the javascript errors contained in your application (unless they directly impact the specific tests causing them to fail).

  • Testing AJAX: WebTest 11 Selenium

Contrary to popular belief, you don’t need to run your test as JavaScript inside a browser to test AJAX functionality. HtmlUnit and thus WebTest is just as well up to the task. It can even be considered superior as it allows better control over how to schedule the in-page requests making the unpredictable browser behavior predictable (see for instance my previous post).

  • Beginner friendly: WebTest 01 Selenium

Beginners (as well as managers ;-)) better understand test automation of web applications, when they see what happens.

  • Documentation: WebTest 10 Selenium

Extensive and up-to-date documentation is very important. A quick look at both web sites will show you that WebTest manual is clearly the winner. It should even be a negative point for Selenium as advice on what makes test suites maintainable is completely missing.

  • Predictable behaviour: WebTest 10 Selenium

Should be a minimal requirement for a test tool, but if you look regularly at Selenium mailing lists or at different posts (like this one), this is not yet fully the case for Selenium.

  • XPath support: WebTest 10 Selenium

WebTest currently uses Jaxen as XPath engine which means that XPath 1.0 is covered as well as some XPath 2 functions (do you know that starts-with is in XPath 1 spec but ends-with first appears in XPath 2?).
Additionally you can customize it to define your own XPath functions.
Selenium uses native XPath support when it’s available (like in Firefox) and evaluates XPath expressions using JavaScript libraries otherwise (like in IE). This JS library is slow and many XPath expressions aren’t interpreted correctly. Even in Firefox, the support is limited to XPath 1.0.

  • Support for badly formatted HTML code: WebTest 01 Selenium

Browsers are able to cope with really badly formed HTML code and so does Selenium as a consequence. WebTest’s parser (NekoHTML) is able to handle some malformations but not that much. Even though it is quite questionable to see it as a feature when your goal is to write your web application as well as possible, sometimes testers do not have access to the development resource to correct the source and just want to test functionality so I’ll give this point to Selenium.

  • Extensibility: WebTest 10 Selenium

Selenium accepts custom extensions but first this is cumbersome because the extensions have to be deployed in the target browser(s) and second interactions are limited as extension code executes in the browser and not in your test program.
In WebTest you have full control over the “browser” from within your tests which you can use to simply write global extensions as well as project or test specific ones.

  • Data driven tests: WebTest 10 Selenium

No discussion, the dataDriven Ant task used with WebTest is simple and powerful!

  • Multi-language support: WebTest 01 Selenium

Selenium RC has bindings in different languages (Java, Ruby, PHP, …) whereas WebTest is bound to Ant which means XML or for instance Groovy with its nice AntBuilder (in fact any of the over 200 languages for the JVM could probably be used). I think that Selenium is missing a real specification language (please don’t talk about Selenese!) like Ant is in this case for WebTest, but I need to give some points to Selenium…

  • Internationalisation support: WebTest 10 Selenium

Using WebTest, you just need to put your language specific strings in property files and use Ant’s built in property task to load the right resources before executing your tests.

Update 05.11.07:
Dan Fabulich correctly indicates that Selenium RC tests written in the same language than the AUT can directly use the application’s i18n resource bundles. Therefore WebTest can’t be seen as winner outside of the Java world.

  • Support for non HTML content: WebTest 10 Selenium

HTML is only one of the content types used by a web application and it’s a common need to have mixed content within the same applications with for instance a PDF file containing the invoice after the checkout. Selenium is limited to HTML content (+XML and text). On the other side WebTest has built in support to work with PDF documents and Excel files as well as Applets and Emails.

Conclusion:

Automated functional tests of web application should become as natural as unit tests. Some tests are better than no tests, no matter which tool is used. Selenium does a good job to introduce newcomers and has many advantages (besides the price) over its commercial model QTP. Nevertheless at least when the size of your suite grows, you should pay attention to your efficiency if you want to last and “the ROI on WebTest is many orders of magnitude higher than any other tool I’ve used” (Lisa Crispin, author of Testing Extreme Programming).

When comparing Selenium and WebTest, 3 categories of web application can be considered. First the applications that are supported both by WebTest and Selenium. This contains most of the applications. Then the ones that use browser features (mostly javascript) that are not yet supported by HtmlUnit. Due to HtmlUnit’s awesome progress in JavaScript support, the size of this category continuously shrinks. The last category concerns applications that use for instance PDF documents or applets and for which Selenium has no support. In all cases where WebTest can be used it is far more efficient than Selenium to ensure the quality of web applications because it provides more feedback and takes less time (both to execute and to analyse results).

When I started writing this post I didn’t expect that Selenium would get such a bad score. Comments are welcome to show which advantages of Selenium over WebTest I’ve missed!

HtmlUnit vs HttpUnit

HtmlUnit committer Daniel Gredler has written an interesting comparison of HtmlUnit and HttpUnit in his blog. I particularly like the analogy with Struts:

“HttpUnit is to the web app testing world what Struts is to the web app framework world: there are many “better” options out there, but it just won’t go away!”

Daniel Gredler blog post: HtmlUnit vs HttpUnit.

Thoughts on Selenium Grid

I’m impressed by the activity around Selenium. Not because I’m impressed by Selenium but because I think that all this work could be better invested. For instance it would surely be a smaller effort to make for instance HtmlUnit able to work with its recognized speed with numerous browser configurations, simulating perfectly their JavaScript execution, and not only 2 browsers as Selenium really supports. The last Selenium related project I’ve discovered is Selenium Grid.

Run tests in parallel

Selenium Grid’s aims to allow running tests on multiple computer because:

  • Selenium remote control is quite slow at driving the browser“.
  • You can only run a limited number of concurrent tests on the same remote control before seriously impacting its stability. Practically speaking, launching more than 6 browsers on the same Selenium Remote Control is not advisable. The limitations are even more drastic for Internet Explorer.

In other words: slow and not scalable. If I correctly understand the documentation, Selenium Grid will allow to run approximatively 6 x N tests in parallel, where N is the number of computers in the grid. Not really impressive.

An HtmlUnit user reported that he performed load testing with HtmUnit using 350 threads in the same VM and one WebClient instance per thread. For this purpose he allocated 512M of his 1Gig laptop to the VM and it worked very well. WebTest surely adds some overhead to HtmlUnit “pure” but, due to the indications of this user, I could imagine that it would be possible to run 500 threads of WebTest tests on one computer within a single VM. This would save just… 80 computers compared to Selenium Grid!

Parallelisation often limited

Selenium Grid’s documentation says: “Selenium Grid cuts down on the time required to run a Selenium test suite to a fraction of the time that a single instance of Selenium instance would take to run“. Sadly this isn’t that easy. Not all tests can be run in parallel: as soon as you test the functionalities of registered users, you’re often limited to a serial test execution (or a just a few parallel executions) when your application doesn’t accept numerous simultaneous login of the same user and you don’t have so much users available with the right settings for the tests. In this case only pure speed can help.

Grid computing is an exciting large area, perhaps is Selenium Grid only a side product of some more significant activity…

Nonsense about HtmlUnit

I’ve just seen the video of Simon Stewart’s Web Driver presentation at last Google Test Automation Conference (GTAC). WebDriver is a bit like jWebUnit but without the assertions bound to the browser interfaces.

Simon says that HtmlUnit is very fast but “doesn’t support JavaScript and the reason why it doesn’t support JavaScript is that HtmlUnit uses the Rhino JavaScript engine (…) and you have to find a browser that uses it“.

Nice explanation… and nobody complained!