Marc Guillemot’s blog

I’ve looked quickly at Mercury’s Quick Test Professional for one year and my impression was that it was a tool with features allowing it to get sold to managers but not for efficient day to day test automation usage.

I’ve discussed recently with a colleague having worked 2 years with QTP and now working since nearly one year with WebTest. I was interested to hear from someone with real QTP experience how the comparison would sound.

He misses 2 QTP features in WebTest:

• the possibility to combine tests of web applications with test in other Window applications (like a test starting with a host connection and going on in a browser)
• the possibility to “see” the responses during the test execution which helps beginners

but considers WebTest as really superior to build reliable, easy to maintain tests of web applications mostly because:

• no time lost for synchronisation configuration during the creation of tests
• native support for XPath
• easy to integrate in continuous integration
• reports provide comprehensive on failed tests
• easily extendible

This mainly confirms my first appreciation: QTP is a tool for people that don’t test!

Perhaps should we go away from Open Source and change WebTest’s license to sell it, let’s say, just half the price of QTP

A JS error with WebTest
During the JAX Ballroom at last JAX conference, I wanted to show to the persons present at the Test Automation table how to get started with WebTest. To do that I chose to write some tests for the conference web site
http://www.jax.de
. Unfortunately WebTest got following error directly at invocation of the url:

The error being located in a js file with “prototype” in its name, I thought that HtmlUnit still had problems with the prototype library although I was convinced that these had been fixed.

Exactly the same with Firefox
On the next day I was motivated to look at it more attentively and was motivated to get it fixed for the presentation we would have on the day after. Indeed the JAX website could be a good example to show some of WebTest features as all attendees should know it. Therefore I started to examine HtmlUnit’s execution of the javascript with a debugger. Unfortunately I couldn’t find anything and first after a while I’ve had a look at the JavaScript console of my Firefox:

WebTest was correct!
This was the explaination: WebTest had no problem and just reacted like Firefox providing precise information concerning the js error (including the line number and file name)!
The faulty line was
for (var i = 0; i < element.childNodes.length; i++) {
and both Firefox and WebTest gave correct message (a bit different) concerning the problem.

It’s possible to tell WebTest to ignore JS errors but it’s not recommanded as it doesn’t help producing high quality Web Sites and http://www.jax.de is an example of WebSite that didn’t get tested correctly (or with a tool that doesn’t help discovering such failures).

Height months after release 2.1 we finally released version 2.5 for 2 days. In fact due to WebTest long tradition of continuous integration, this is “just” build 1555 which has only minor improvements over build 1554, which… Why a release in such a case? In fact 2 reasons: first many users prefer to use a release rather than some other build and second this is the occasion to communicate about our preferred web testing tool and to tell why it is the most efficient way to test a web app.

Why a jump from 2.1 to 2.5?

This was mainly motivated by the very important changes performed in WebTest internals which have only small visible effects (smaller memory consumption, seamless integration of macros or external target calls within <steps>…</steps>).

What’s special in this release?

Besides the internal changes, the main changes are probably:

• the upgrade to htmlunit 1.11 allowing a far better javascript support
• the new selectWebClient step allowing to drive more than one web client within a single test (particularly useful when you have a scenario involving users with different roles that should interact)
• the xpath attribute for the repeat step
• the new ready to import webtest.xml build file
• and of course diverse improvements in the reports

What’s next?

Difficult to say as changes often come when the need arise. Many AJAX related improvements will probably come like a way to synchronize some asynchronous calls to make them testable without sleep or waitFor… Look at the mailing list to stay up to date.