HtmlUnit 2.8 released

Nearly 6 months after release 2.7, HtmlUnit-2.8 is finally ready.

This release comes later as originally planned as we wanted to avoid as most as possible any side effect of the large internal changes. I’m now quite confident that we have reached a really good status.

Here is an extract of the change log:

  • single thread for background JS execution
  • migration to HttpClient 4
  • initial support for FF3.6 simulation
  • support for SOCKS proxy
  • initial support for Google AppEngine
  • support for large (binary) download

Download: http://sourceforge.net/project/showfiles.php?group_id=47038

The support for Google AppEngine which is based on contributions from Google’s GWT team is exciting as it will open the door to new usage of HtmlUnit.

The number of HtmlUnit users continuously grows and, as the consequence, so does the number of reported issues and feature wishes. We will try to address it in next release.

WebTest with Groovy, Maven and Eclipse

WebTest is in fact “only” a set of Ant tasks therefore it’s not a surprise that WebTest tests are traditionally written in XML. Grails WebTest plugin has shown that Groovy can be a very pleasant alternative thanks to the AntBuilder. Even if you don’t use Grails, you can write WebTest tests in Groovy using the support provided by WebTest’s distribution but if you plan to write all your tests in Groovy, this is surely not optimal. This post shows step by step a way to write WebTest tests that is near to a traditional setup for unit tests using Groovy and Maven.

Note: I’m not a Maven expert. The proposed code works but it can perhaps be improved.

  1. Create a new Maven project
    mvn archetype:create -DarchetypeGroupId=org.apache.maven.archetypes -DgroupId=my.domain -DartifactId=myWebTestApp
    

  2. Edit the generated pom.xml
    1. Add WebTest as dependency:
          <dependencies>
              ...
              <dependency>
                  <groupId>com.canoo.webtest</groupId>
                  <artifactId>webtest</artifactId>
                  <version>3.1-SNAPSHOT</version>
              </dependency>
          </dependencies>
      
    2. Add reference to WebTest’s Maven snapshot repository
      	    <repositories>
      	        <repository>
      	            <id>webtest_dependencies_snapshot</id>
      	            <name>WebTest dependencies</name>
      	            <url>http://webtest.canoo.com/webtest/m2-repo-snapshots</url>
      	        </repository>
      	    </repositories>
      

      Note that this is needed only if you want to use the latest WebTest snapshot (which is always the best one). You don’t need it if you use a version that is available on the central repository. As of Apr. 30, WebTest 3.0 has not been uploaded to the central repository. The upload request is available here: MAVENUPLOAD-2439.

      Update: WebTest 3.0 is available in Maven central repository since May 19.

    3. Configure the GMaven plugin
      <build>
      	<plugins>
      		<plugin>
      			<groupId>org.codehaus.groovy.maven
      			</groupId>
      			<artifactId>gmaven-plugin</artifactId>
      			<version>1.0-rc-5</version>
      			<executions>
      				<execution>
      					<goals>
      						<goal>compile</goal>
      						<goal>testCompile</goal>
      					</goals>
      				</execution>
      			</executions>
      		</plugin>
      		<plugin>
      			<groupId>org.codehaus.mojo</groupId>
      			<artifactId>build-helper-maven-plugin</artifactId>
      			<executions>
      				<execution>
      					<id>add-test-source</id>
      					<phase>generate-sources</phase>
      					<goals>
      						<goal>add-test-source</goal>
      					</goals>
      					<configuration>
      						<sources>
      							<source>src/test/groovy</source>
      						</sources>
      					</configuration>
      				</execution>
      			</executions>
      		</plugin>
      	</plugins>
      </build>
      
  3. Create the Eclipse project
    mvn -Declipse.downloadSources=true eclipse:eclipse
    

    I guess that something similar exists for other IDEs.

  4. Import the project in your workspace
  5. Add Groovy support in the IDE
    The Groovy support in Eclipse is not as good as in other IDEs but it is better than nothing.
  6. Write your first WebTest in Groovy src/test/groovy/my/domain/FirstWebTest.groovy
    package my.domain
    
    import com.canoo.webtest.WebtestCase
    
    /**
     * My first WebTest in Groovy.
     */
    class FirstWebTest extends WebtestCase {
    
    	void testGoogleSearch() {
    		webtest("My first test") {
    			invoke "http://www.google.com/ncr", description: "Go to Google (in English)"
    			verifyTitle "Google"
    			setInputField name: "q", value: "WebTest"
    			clickButton "I'm Feeling Lucky"
    			verifyTitle "Canoo WebTest"
    		}
    	}
    }
    
  7. Run the test
    • As normal unit test from the IDE
      As it is a normal unit test and you can use the IDE support for that. In Eclipse this can be done with right mouse click / Run As... / Junit Test or with the shortcut Alt + Shift + X Z
    • Or from the command line
      mvn test
      
  8. Enjoy the results!
    Running a WebTest this way produces “normal” JUnit reports as well as the traditional reports of WebTest that contain precise information to quickly find the cause of failed tests.
  9. (optional) Run headless
    Per default WebTest shows a console that informs about the running tests and opens the reports in your browser once the tests are finished. This is often useful on a workstation but it may be really annoying on a build server. To avoid that, you just need to set the wt.headless property:

    mvn test -Dwt.headless
    

HtmlUnit: full support for MochiKit

Upcoming release 2.4 of HtmlUnit will provide 100% support (1) for the JavaScript library MochiKit (version 1.4.1)!

The library’s own tests are integrated in HtmlUnit’s build and they now all (2) pass, no matter if we simulate Firefox 2, Firefox 3, Internet Explorer 6 or Internet Explorer 7. This time, it was not so complicated to reach 100% support: fix a bug in Rhino, implement a dummy support for the filters property on HTML elements when simulating IE, handle detached nodes for offsetParent, and fix a few things in computing CSS properties.

I believe that we’re on the right way and that we’ll soon be able to announce full support for other well known libraries.

(1) I suppose here that the library’s own tests are representative for the library usage.
(2) In fact we run only the tests files that pass in a “real” browser and therefore have to ignore 3 files.

WebTest @ JUG Cologne slides

Here are the slides of the presentation I’ve made yesterday:

Yet Another JRuby Wrapper for HtmlUnit: schnell

After Celerity, a new open source JRuby library, schnell, uses HtmlUnit to provide a fast Watir-like API for functional testing of web applications. According to the announcement, it runs… 37,5 times faster than Watir!

(thanks Daniel for the info)

WebTest/HtmlUnit integration with JMeter started

I couldn’t resist to show this screenshot before my holiday:

HtmlUnit in .NET and HtmlUnit in JRuby with Celerity as fast compatible alternative to Watir

Java and Groovy are my main programming languages (as well as JavaScript for scripts in HTML pages) therefore it’s always funny for me to see that HtmlUnit gets used outside my “Java world”.

HtmlUnit in .NET

HtmlUnit is a Java library but its usage seems not limited to the JVM. For some time I found the blog post “Some Goodies: Model-View-Presenter Article, Java in .NET, HtmlUnit” where the author describes how he successfully used IKVM to use HtmlUnit from his .NET environment. Quite funny.

HtmlUnit in JRuby: Celerity as faster Watir

The initial setup for the second example was surely simpler as it runs on the JVM. The Norwegian provider FINN.no had performance problems with its Watir test suite completing in 3 hours which is not very agile. To improve this situation, they decided to create the Open Source project Celerity to provide a Watir-compatible API wrapping HtmlUnit to improve execution speed without to rewrite all their tests. It would be interesting to know how fast their suite now runs and if they use new possibilities of this approach like running tests in parallel.

The Celerity project perfectly demonstrates that speed matters. You can frequently find “experts” writing that functional tests are slow per nature. This is fully wrong, functional tests won’t be as fast as unit tests but HtmlUnit and derived tools (like Celerity, WebDriver with HtmlUnit driver, WebTest) demonstrate that functional tests can be quite fast.

.NET, JRuby, and then?

The interest for HtmlUnit continually grows and I’m curious to see what will be the next “exotic” usage of HtmlUnit. Don’t hesitate to communicate your experiences.

Boost your WebTests: 50% faster or more!

Tests never run fast enough!

This is particularly true for functional tests of web applications. WebTest is known to be very fast compared to other functional test tools, nevertheless when your test suite grows or simply when you want to quickly receive feedback after a commit, you often feel that tests are too slow. A new experimental feature of WebTest allows to specify the number of threads that should be used for the tests what can bring enormous speed improvements without modification of the tests.

Simply configure the number of threads

Rather than just calling

ant

you just specify the number of worker threads

ant -Dwt.parallel.nbWorkers=20

The optimal number of worker threads will depend from the usage.

How it works: enqueue rather than execute

The idea is quite simple and the implementation totally non-intrusive. In fact this is a shame that nobody has had the idea previously (including you! ;-)). This can be simply explained with a few lines of code.

WebTest is based on Ant what means that the task mapped to <webtest> contains something like this:

class WebTestTask extends Task
{
  void execute()
  {
     // do the WebTest
  }
}

For the parallel execution we map <webtest> to a new class that looks like this:

class WebTestTaskParallel extends WebTestTask
{
	void execute()
	{
		workQueue.add this
	}

	void executeReally()
	{
		super.execute()
	}
}

Once this is done, Ant can do its job normally. When it calls the execute() method of the task, the instance adds itself to a queue and the execute() method returns allowing Ant to continue its normal execution. A set of worker threads look at the queue, calling executeReally() on the WebTestTask instances to really run the tests. That’s nearly everything, the rest is just synchronization glue code.

Improvements from 0% to 50% and more

I’ve seen many cases where using a few threads allows to gain over 50% execution speed. In fact this will depend from the capacity of the application under test and how it can handle the increased charge. With a server – not necessary fast – that smoothly handles the charge generated by numerous worker threads, I can imagine tests suites that execute approximatively in the time of the slowest test. On the other side I’ve already seen test suites that take more time to execute with many worker threads when the AUT runs on the same physical machine than the tests themselves. It’s not really surprising: if the tests need more “power” to execute, then there is less for the application.

Limitations

With a recent WebTest build, you just have to specify the number of worker threads to start using this new feature. You many need to refactor your tests to include processing made after a </webtest> within the test otherwise this will be done before the tests itself gets executed and may be the cause of problems. Finally current implementation of this feature doesn’t contain any facility to manage dependencies between tests.

Towards functional load testing?

Tests with large numbers of worker threads have shown that WebTest scales quite well: both memory and CPU usage stay correct. This opens a new perspective for WebTest: the usage as load testing tool. In fact it should probably be bound to something like JMeter but I see great advantages in WebTest usage compared to low level http processing as usually done by load testing tools, particularly in test maintainability, AJAX testing or for the correctness of the simulated scenario.

Update (Feb 6): a former client told me that they just started to use a slightly modified version of this feature I prepared them for some time (allowing to manage some dependencies between tests). The results are even better as expected: over 75% saved time! With 8 working threads the tests complete in less than 25% of the original time.

WebTest as universal DSL for automated web testing in Groovy thanks to its Ant roots?

What for a long title!

In the discussion in Grails-user mailing list following my previous post “WebTest vs Selenium“, Marc Palmer and James Page requested the creation of a kind of meta DSL in Groovy for automated functional tests of web applications.

The idea was to provide a DSL allowing to write functional tests in a tool agnostic way and to run them with WebTest as often as needed because it is fast and for instance once each night using Selenium because it uses a real browser but is quite slow.

I’m still not fully convinced on the utility of such a feature because WebTest is so good 😉 but I think that there is no need for a new DSL: it already exists! The AntBuilder allows WebTest to have a really nice syntax in Groovy and this could be simply reused for other “target tools” thanks to Ant.

Ant theory

When you write something like this (the examples in this post use Groovy AntBuilder but the same would apply for “pure” Ant in XML format):

ant.webtest(name: "a simple test)
{
  invoke "http://webtest.canoo.com"
  verifyTitle "WebTest website"
  clickLink "Manual"
  verifyXPath(xpath: "count(id('navigation-top')//li)", text: "5",
      description: "check the number of top menu elements")
  ...
}

this looks like WebTest but this is not WebTest as long as you haven’t configured Ant with for instance something like

ant.project.addTaskDefinition("webtest", com.canoo.webtest.ant.WebTestTask)
ant.project.addTaskDefinition("invoke", com.canoo.webtest.steps.request.Invoke)
...

This has 2 consequences:
– it is possible to configure other tasks than WebTest’s ones for “WebTest steps”
– it is possible to inspect the “parsed” tasks tree

This means that it wouldn’t be too difficult to run a “WebTest” test with an other automated test tool like for instance Selenium (as long as the tests don’t use any WebTest feature for which no Selenium equivalent exists). Let’s see how this could be done.

First solution: redefine “WebTest steps”

The first approach consists in redefining the “WebTest steps” to provide an alternative implementation for each WebTest step before executing the test. This could look like following for verifyTitle:

import org.apache.tools.ant.*class SeleniumVerifyTitle extends Task
{
  String text
  void execute()
  {
    if (text != selenium.title)
      throw new BuildException("Wrong title: expected $text, got ${selenium.title}")
  }
  def getSelenium()
  {
    project.references.'selenium' // assuming that test start placed it there
  }
}
ant.project.taskDefinitions["verifyTitle"] = SeleniumVerifyTitle

verifyTitle is quite simple, for other steps it would be more tricky or even impossible (like the pdf or email steps) to write a Selenium equivalent.

Generate script from Ant tree

The second approach consist in the generation of a script from the Ant structure. This has the “advantage” that the
generated script doesn’t necessarily have to be run on the Java Virtual Machine.

class WebTest2SeleniumRubyConverter extends WebTest
{
  File targetFile // additional attribute to Ant task: the file to write in
  private converters = [
    'invoke': { "open \"${it.attributeMap.url}\"" },
    'verifyTitle': { "assert_equal \"${it.attributeMap.text}\", @selenium.get_title" },
    'clickLink': { "@selenium.click \"link=${it.attributeMap.label}\"" },
    'verifyXPath': { "# ?? I haven't found an example" },
    'pdfVerifyTitle': { "# skipped because not supported: pdfVerifyTitle ${it.attributeMap.title}" },
    ...
  ]
  def execute()
  {
    def rubySteps = runtimeConfigurableWrapper.children.collect { converters[it.elementTag](it) }
    def scriptTemplate = """
require 'test/unit'
require 'selenium'
class ExampleTest < Test::Unit::TestCase
include SeleniumHelper

def setup
@selenium = Selenium::SeleniumDriver.new("localhost", 4444, "*firefox", "http://localhost", 10000);
@selenium.start
end

def teardown
@selenium.stop
end

def test_something
<% steps.each { %>
$it
<% }%>
end

end"""

    def engine = new groovy.text.SimpleTemplateEngine()
    def template = engine.createTemplate(scriptTemplate)
    targetFile.withWriter {
      it << template.make([steps: rubySteps])
    }
  }
}
ant.project.taskDefinitions["webTest"] = WebTest2SeleniumRubyConverter

NB: I don’t have any experience in Ruby and have written the example code above only by looking at the samples on Selenium’s website.

Naturally this is only an example and it is a bit more complicated to do it correctly (handle different combinations of task attributes, special characters, Ant macros, …) but surely not so much.

Conclusion: not very complicated but does it makes sense?

These two examples shows that it wouldn’t be complicated to “convert” simple WebTest tests to allow them to be executed with an other tool like Selenium (but other tools like for instance WebDriver could be a target too) as long as they use the subset of WebTest features that the target tool accepts. Of course WebTest features like its particularly rich reporting wouldn’t be available either.

Personally I’m not really convinced of the utility of such a “generic DSL” due to WebTest’s excellent quality and unless a new client (you? ;-)) really wants this feature I don’t plan to work on it but I’m ready to provide technical assistance if someone wants to do the job.

2nd WebTest screencast: Data Driven WebTest

My colleague Tomi Schütz and I have finished our second WebTest screencast: Data driven WebTest.

Content: (duration ~2’30”)
I- Do you know Google Calculator?
II- Do you know the dataDriven Ant task?
III- Testing Google Calculator with data driven WebTest

It should be interesting even for experienced WebTest users: the dataDriven task is so simple and powerful, and Google Calculator is a funny toy.

« Older entries