Boost your WebTests: 50% faster or more!

February 4, 2008 at 6:54 pm (Test Automation, WebTest)

Tests never run fast enough!

This is particularly true for functional tests of web applications. WebTest is known to be very fast compared to other functional test tools, nevertheless when your test suite grows or simply when you want to quickly receive feedback after a commit, you often feel that tests are too slow. A new experimental feature of WebTest allows to specify the number of threads that should be used for the tests what can bring enormous speed improvements without modification of the tests.

Simply configure the number of threads

Rather than just calling

ant

you just specify the number of worker threads

ant -Dwt.parallel.nbWorkers=20

The optimal number of worker threads will depend from the usage.

How it works: enqueue rather than execute

The idea is quite simple and the implementation totally non-intrusive. In fact this is a shame that nobody has had the idea previously (including you! ;-)). This can be simply explained with a few lines of code.

WebTest is based on Ant what means that the task mapped to <webtest> contains something like this:

class WebTestTask extends Task
{
  void execute()
  {
     // do the WebTest
  }
}

For the parallel execution we map <webtest> to a new class that looks like this:

class WebTestTaskParallel extends WebTestTask
{
	void execute()
	{
		workQueue.add this
	}

	void executeReally()
	{
		super.execute()
	}
}

Once this is done, Ant can do its job normally. When it calls the execute() method of the task, the instance adds itself to a queue and the execute() method returns allowing Ant to continue its normal execution. A set of worker threads look at the queue, calling executeReally() on the WebTestTask instances to really run the tests. That’s nearly everything, the rest is just synchronization glue code.

Improvements from 0% to 50% and more

I’ve seen many cases where using a few threads allows to gain over 50% execution speed. In fact this will depend from the capacity of the application under test and how it can handle the increased charge. With a server – not necessary fast – that smoothly handles the charge generated by numerous worker threads, I can imagine tests suites that execute approximatively in the time of the slowest test. On the other side I’ve already seen test suites that take more time to execute with many worker threads when the AUT runs on the same physical machine than the tests themselves. It’s not really surprising: if the tests need more “power” to execute, then there is less for the application.

Limitations

With a recent WebTest build, you just have to specify the number of worker threads to start using this new feature. You many need to refactor your tests to include processing made after a </webtest> within the test otherwise this will be done before the tests itself gets executed and may be the cause of problems. Finally current implementation of this feature doesn’t contain any facility to manage dependencies between tests.

Towards functional load testing?

Tests with large numbers of worker threads have shown that WebTest scales quite well: both memory and CPU usage stay correct. This opens a new perspective for WebTest: the usage as load testing tool. In fact it should probably be bound to something like JMeter but I see great advantages in WebTest usage compared to low level http processing as usually done by load testing tools, particularly in test maintainability, AJAX testing or for the correctness of the simulated scenario.

Update (Feb 6): a former client told me that they just started to use a slightly modified version of this feature I prepared them for some time (allowing to manage some dependencies between tests). The results are even better as expected: over 75% saved time! With 8 working threads the tests complete in less than 25% of the original time.