WebTest/HtmlUnit integration with JMeter started

I couldn’t resist to show this screenshot before my holiday:

Oudated knowledge of “experts”: the example of HttpUnit

If you look at the download statistics of HtmlUnit and HttpUnit at Sourceforge (here for HtmlUnit and here for HttpUnit), you’ll see that HttpUnit still has more downloads than HtmlUnit. This is quite surprising when you have some knowledge of both libraries and particularly of the fact that HttpUnit is deeply asleep since so many years (the recent 1.7 release doesn’t really change the deal).

In his blog post “HtmlUnit vs HttpUnit“, Daniel Gredler wrote “HttpUnit is to the web app testing world what Struts is to the web app framework world: there are many “better” options out there, but it just won’t go away!“. I totally agree but it is interesting to examine the reasons why it doesn’t go away. I believe that one element of the answer is that many “experts” have outdated knowledge in this domain and still talk about HttpUnit.

Some examples:

  • Rod Johnson

In “System Integration Testing Using Spring” at the The Spring Experience conference, Rod talked about HttpUnit but not about HtmlUnit.

  • JSFUnit

The JBoss project JSFUnit contains HttpUnit not HtmlUnit.

Nevertheless it seems that in this case the mistake has been identified as a feature request “Investigate replacing HttpUnit with HtmlUnit” already exists.

  • Recent books

Different recent books mention HttpUnit but not HtmlUnit, like for instance Java Power Tools (by John Ferguson Smart), Test Driven, Practical TDD and Acceptance TDD for Java Developers (by Lasse Koskela), or Ant in Action, 2nd Edition (by Steve Loughran and Erik Hatcher). In this latest case, rather than HtmlUnit, they should probably have written about WebTest as it is probably the tool – web testing or not – that makes the most intensive usage of Ant.

Why?

Why did all these famous developers talk / write about HttpUnit? My guess is simply that functional tests of web applications are not their favorite field of expertise and therefore that they didn’t update their knowledge in this area since a – too – long time.

The consequences

I see two consequences. First it gives a bad impression on the whole presented content: no matter how good it may be, when you detect outdated areas, you start to question the rest. Second, it influences newcomers that have no idea of the domain and this can be a good explaination for HttpUnit surprising high download numbers.

AJAX and load testing: HtmlUnit as a solution?

If you search for a load testing tool to test your AJAX application, you will surely find many vendors pretending that their tool support AJAX applications. If you look a bit more carefully in the detailed features, you’ll quickly find that it is done by recording the http traffic.

HTTP traffic recording limits

Capturing should always be considered very carefully for functional testing as well as for load testing. In the case of AJAX applications, it can simply lead you to generate a load testing scenario that may work… but that has nothing to do with the traffic that would really occur for the number of simulated users.

A small example

An example is worth 1000 words, let’s look at the code of this minimal AJAX application

<html><head><title>Welcome</title><body>
<script>
var xhr = window.XMLHttpRequest ? new XMLHttpRequest() 
  : new ActiveXObject('Microsoft.XMLHTTP');
xhr.open('GET', 'numberOfVisitors.txt', false);
xhr.send( "" )

var nbConnectedVisitors = parseInt(xhr.responseText)
var nextPage;
if (nbConnectedVisitors < 100)
	nextPage = "fullFeaturedSite.html"
else if (nbConnectedVisitors < 1000)
	nextPage = "limitedSite.html"
else
    nextPage = "temporarilyUnavailable.html"
	
window.location = nextPage
</script>
</body></html>

It is extremely simple: the first html page uses an XMLHttpRequest to ask the server for the number of currently connected users. Client side javascript logic is used to determine the page to show next, depending on the current charge on the server: “fullFeaturedSite.html” if less than 100 users are connected, “limitedSite.html” for 100 until 1000 connected users and “temporarilyUnavailable.html” when over 1000 users are connected.
If you use a recorder to create a test for this application, you’ll probably be the single connected user while recording. When you’ll run the recorded tests to simulate the last of a large number of users, you’ll perform charge on your server of course (here requesting “fullFeaturedSite.html“), but this charge is something that would never occur when over 100 users are connected, what means that your test results have nearly no signification.

Client side JS evaluation needed

It wouldn’t be really complicated here to adapt the load script to parse the content of the response from “numberOfVisitors.txt” in order to perform the adequate next request. The downside of such a strategy is naturally the duplication of the logic of the application in the load test scripts which cause high maintenance costs and is not necessarily always as easy as in this oversimplified example.

A place for HtmlUnit as load testing tool?

Load testing tools are able to simulate a large number of simultaneous users because they are low level and perform only minimal processing of the server responses. Visual web browsers on the other side can interpret the JavaScript code on the visited web sites but are too heavy weight to be used to produce load unless you have a very large number of computers available for the exercise. HtmlUnit could play an interesting role between these two extrema: it is lightweight enough to allow to simulate a few 100 users on a single machine and it has already an excellent support for JavaScript, allowing it to behave just like a “normal” browser for numerous AJAX applications. As far as I know, HtmlUnit is unique in this category therefore I believe that its usage for load testing will grow in the future months.

HtmlUnit in .NET and HtmlUnit in JRuby with Celerity as fast compatible alternative to Watir

Java and Groovy are my main programming languages (as well as JavaScript for scripts in HTML pages) therefore it’s always funny for me to see that HtmlUnit gets used outside my “Java world”.

HtmlUnit in .NET

HtmlUnit is a Java library but its usage seems not limited to the JVM. For some time I found the blog post “Some Goodies: Model-View-Presenter Article, Java in .NET, HtmlUnit” where the author describes how he successfully used IKVM to use HtmlUnit from his .NET environment. Quite funny.

HtmlUnit in JRuby: Celerity as faster Watir

The initial setup for the second example was surely simpler as it runs on the JVM. The Norwegian provider FINN.no had performance problems with its Watir test suite completing in 3 hours which is not very agile. To improve this situation, they decided to create the Open Source project Celerity to provide a Watir-compatible API wrapping HtmlUnit to improve execution speed without to rewrite all their tests. It would be interesting to know how fast their suite now runs and if they use new possibilities of this approach like running tests in parallel.

The Celerity project perfectly demonstrates that speed matters. You can frequently find “experts” writing that functional tests are slow per nature. This is fully wrong, functional tests won’t be as fast as unit tests but HtmlUnit and derived tools (like Celerity, WebDriver with HtmlUnit driver, WebTest) demonstrate that functional tests can be quite fast.

.NET, JRuby, and then?

The interest for HtmlUnit continually grows and I’m curious to see what will be the next “exotic” usage of HtmlUnit. Don’t hesitate to communicate your experiences.