I couldn’t resist to show this screenshot before my holiday:
If you look at the download statistics of HtmlUnit and HttpUnit at Sourceforge (here for HtmlUnit and here for HttpUnit), you’ll see that HttpUnit still has more downloads than HtmlUnit. This is quite surprising when you have some knowledge of both libraries and particularly of the fact that HttpUnit is deeply asleep since so many years (the recent 1.7 release doesn’t really change the deal).
In his blog post “HtmlUnit vs HttpUnit“, Daniel Gredler wrote “HttpUnit is to the web app testing world what Struts is to the web app framework world: there are many “better” options out there, but it just won’t go away!“. I totally agree but it is interesting to examine the reasons why it doesn’t go away. I believe that one element of the answer is that many “experts” have outdated knowledge in this domain and still talk about HttpUnit.
- Rod Johnson
In “System Integration Testing Using Spring” at the The Spring Experience conference, Rod talked about HttpUnit but not about HtmlUnit.
The JBoss project JSFUnit contains HttpUnit not HtmlUnit.
Nevertheless it seems that in this case the mistake has been identified as a feature request “Investigate replacing HttpUnit with HtmlUnit” already exists.
- Recent books
Different recent books mention HttpUnit but not HtmlUnit, like for instance Java Power Tools (by John Ferguson Smart), Test Driven, Practical TDD and Acceptance TDD for Java Developers (by Lasse Koskela), or Ant in Action, 2nd Edition (by Steve Loughran and Erik Hatcher). In this latest case, rather than HtmlUnit, they should probably have written about WebTest as it is probably the tool – web testing or not – that makes the most intensive usage of Ant.
Why did all these famous developers talk / write about HttpUnit? My guess is simply that functional tests of web applications are not their favorite field of expertise and therefore that they didn’t update their knowledge in this area since a – too – long time.
I see two consequences. First it gives a bad impression on the whole presented content: no matter how good it may be, when you detect outdated areas, you start to question the rest. Second, it influences newcomers that have no idea of the domain and this can be a good explaination for HttpUnit surprising high download numbers.
If you search for a load testing tool to test your AJAX application, you will surely find many vendors pretending that their tool support AJAX applications. If you look a bit more carefully in the detailed features, you’ll quickly find that it is done by recording the http traffic.
HTTP traffic recording limits
Capturing should always be considered very carefully for functional testing as well as for load testing. In the case of AJAX applications, it can simply lead you to generate a load testing scenario that may work… but that has nothing to do with the traffic that would really occur for the number of simulated users.
A small example
An example is worth 1000 words, let’s look at the code of this minimal AJAX application
<html><head><title>Welcome</title><body> <script> var xhr = window.XMLHttpRequest ? new XMLHttpRequest() : new ActiveXObject('Microsoft.XMLHTTP'); xhr.open('GET', 'numberOfVisitors.txt', false); xhr.send( "" ) var nbConnectedVisitors = parseInt(xhr.responseText) var nextPage; if (nbConnectedVisitors < 100) nextPage = "fullFeaturedSite.html" else if (nbConnectedVisitors < 1000) nextPage = "limitedSite.html" else nextPage = "temporarilyUnavailable.html" window.location = nextPage </script> </body></html>
fullFeaturedSite.html” if less than 100 users are connected, “
limitedSite.html” for 100 until 1000 connected users and “
temporarilyUnavailable.html” when over 1000 users are connected.
If you use a recorder to create a test for this application, you’ll probably be the single connected user while recording. When you’ll run the recorded tests to simulate the last of a large number of users, you’ll perform charge on your server of course (here requesting “
fullFeaturedSite.html“), but this charge is something that would never occur when over 100 users are connected, what means that your test results have nearly no signification.
Client side JS evaluation needed
It wouldn’t be really complicated here to adapt the load script to parse the content of the response from “
numberOfVisitors.txt” in order to perform the adequate next request. The downside of such a strategy is naturally the duplication of the logic of the application in the load test scripts which cause high maintenance costs and is not necessarily always as easy as in this oversimplified example.
A place for HtmlUnit as load testing tool?
HtmlUnit in .NET
HtmlUnit is a Java library but its usage seems not limited to the JVM. For some time I found the blog post “Some Goodies: Model-View-Presenter Article, Java in .NET, HtmlUnit” where the author describes how he successfully used IKVM to use HtmlUnit from his .NET environment. Quite funny.
HtmlUnit in JRuby: Celerity as faster Watir
The initial setup for the second example was surely simpler as it runs on the JVM. The Norwegian provider FINN.no had performance problems with its Watir test suite completing in 3 hours which is not very agile. To improve this situation, they decided to create the Open Source project Celerity to provide a Watir-compatible API wrapping HtmlUnit to improve execution speed without to rewrite all their tests. It would be interesting to know how fast their suite now runs and if they use new possibilities of this approach like running tests in parallel.
The Celerity project perfectly demonstrates that speed matters. You can frequently find “experts” writing that functional tests are slow per nature. This is fully wrong, functional tests won’t be as fast as unit tests but HtmlUnit and derived tools (like Celerity, WebDriver with HtmlUnit driver, WebTest) demonstrate that functional tests can be quite fast.
.NET, JRuby, and then?
The interest for HtmlUnit continually grows and I’m curious to see what will be the next “exotic” usage of HtmlUnit. Don’t hesitate to communicate your experiences.