WebTest vs Selenium: WebTest wins 13 – 5

In the last months I’ve seen a rising interest in automated testing of web applications thanks to the efficient viral marketing of Selenium. However, the world is full of test automation projects that started with big hopes and lots of enthusiasm only to be abandoned shortly after facing the unpleasant reality that it needs more than a point-and-click activity to develop a suite of robust tests.

The maintainability of automated tests depends primarily on the skills of the test authors, but different tools have different features that impact their efficiency. This blog post compares the features of two open source automated web testing tools: Canoo WebTest and Selenium.

A short introduction to the contenders:

Canoo WebTest is a free open source tool that has existed since 2001. It is written in pure Java and contains a set of Ant tasks that drive a simulated, faceless browser (originally HttpUnit, but for the last few years HtmlUnit).

Selenium is a free open source tool as well, created in 2004. It uses injected JavaScript to work within real browsers. Different components exists under the name Selenium: Selenium Core, Selenium RC, Selenium IDE (!), … In this blog post I will only consider Selenium RC used with Selenium IDE on Firefox or Selenium HTA on Internet Explorer due to the limitations of the other possibilities.

Features comparison

To be clear, as a WebTest (and HtmlUnit) committer I’m undoubtedly biased. On the other hand, I have experience with huge functional test suites being developed and maintained over periods of years. Trying to be objective, I may overcompensate in the other direction and give Selenium too much credit. Of course I will diligently fix errors I may have in my Selenium understanding. But please read this post until the end before starting with criticisms ;-)

  • Browser fidelity: WebTest 01 Selenium

This is probably the most overestimated characteristic of a web testing tool. Automated tests don’t make manual testing useless because automated tests can’t cover everything (at least for affordable costs). You still have to walk through your application (just think of everything you’ve checked just reading this: page layout, font size, font colors, …).
The consequence is that an automated web testing tool’s purpose is not to ensure that an application works “well” as it is not possible, but to detect most of the failures that could happen. This is a huge difference because it means that tests don’t have necessary to run in a “real” browser.
Nevertheless the browser’s real behaviour has to be approximated as closely as possible. HtmlUnit’s JavaScript support has made impressive progress but it still doesn’t (and will never) behave exactly like a normal browser.
Even though Selenium modifies the normal JavaScript execution of an application, it uses the browser itself and therefore is nearer to the standard behaviour of the browser.

  • Reports: WebTest 10 Selenium

JUnit-like reports are far too limited for web test automation. This is probably something that you first see when you have reached a certain volume of tests. If the tests are successful, you don’t need any report at all, but when some tests fail, you need the information to find as quickly as possible what is the failure cause and an error message is often not enough.
With comprehensive reports like those provided by WebTest, you don’t have to debug your tests, just to analyse the reports. Furthermore it allows you to understand (and fix) the worst kind of bugs: those that don’t occur systematically.

  • Speed: WebTest 10 Selenium

Tests are never fast enough. Selenium is known not to be very fast and even slower on Internet Explorer (just read the mailing list) and seems to suffer from memory leaks. On the other side, WebTest is quite fast (see for instance this thread in Selenium Dev mailing list for a non representative test where Selenium took ~ 10 seconds and WebTest < 2s).
It’s not surprising due to Selenium’s architecture (3 tiers involved) and all the rendering that happens in browser. Even if HtmlUnit’s HTML handling algorithms are not as good as the real browsers, WebTest has simply less to do and everything happens in the JVM.

  • Integration in development process: WebTest 10 Selenium

WebTest is “just” Ant which means that it can directly be called from CruiseControl for instance or from each developer’s workstation.
On the other side for Selenium you need a real browser with an own profile, a proxy – possibly on another computer if you want to test with IE and run the tests from a non Windows system.

  • Scalability: WebTest 10 Selenium

For a large application (or set of applications) with a good functional test coverage your test suite(s) will rapidly grow and scalability may become an issue. WebTest scales far better than Selenium mostly because it’s faster and because you can simply run many tests suites in parallel (just think of the hardware requirements and browser limitations to do that with Selenium).

  • Capture JS errors: WebTest 10 Selenium

This is what surprises me most about experienced developers working with Selenium: they find it acceptable to ignore JS errors. Would you accept compilation errors in your program as long as your unit test pass? Surely not! But in fact this is exactly what you do with Selenium as it doesn’t detect the javascript errors contained in your application (unless they directly impact the specific tests causing them to fail).

  • Testing AJAX: WebTest 11 Selenium

Contrary to popular belief, you don’t need to run your test as JavaScript inside a browser to test AJAX functionality. HtmlUnit and thus WebTest is just as well up to the task. It can even be considered superior as it allows better control over how to schedule the in-page requests making the unpredictable browser behavior predictable (see for instance my previous post).

  • Beginner friendly: WebTest 01 Selenium

Beginners (as well as managers ;-)) better understand test automation of web applications, when they see what happens.

  • Documentation: WebTest 10 Selenium

Extensive and up-to-date documentation is very important. A quick look at both web sites will show you that WebTest manual is clearly the winner. It should even be a negative point for Selenium as advice on what makes test suites maintainable is completely missing.

  • Predictable behaviour: WebTest 10 Selenium

Should be a minimal requirement for a test tool, but if you look regularly at Selenium mailing lists or at different posts (like this one), this is not yet fully the case for Selenium.

  • XPath support: WebTest 10 Selenium

WebTest currently uses Jaxen as XPath engine which means that XPath 1.0 is covered as well as some XPath 2 functions (do you know that starts-with is in XPath 1 spec but ends-with first appears in XPath 2?).
Additionally you can customize it to define your own XPath functions.
Selenium uses native XPath support when it’s available (like in Firefox) and evaluates XPath expressions using JavaScript libraries otherwise (like in IE). This JS library is slow and many XPath expressions aren’t interpreted correctly. Even in Firefox, the support is limited to XPath 1.0.

  • Support for badly formatted HTML code: WebTest 01 Selenium

Browsers are able to cope with really badly formed HTML code and so does Selenium as a consequence. WebTest’s parser (NekoHTML) is able to handle some malformations but not that much. Even though it is quite questionable to see it as a feature when your goal is to write your web application as well as possible, sometimes testers do not have access to the development resource to correct the source and just want to test functionality so I’ll give this point to Selenium.

  • Extensibility: WebTest 10 Selenium

Selenium accepts custom extensions but first this is cumbersome because the extensions have to be deployed in the target browser(s) and second interactions are limited as extension code executes in the browser and not in your test program.
In WebTest you have full control over the “browser” from within your tests which you can use to simply write global extensions as well as project or test specific ones.

  • Data driven tests: WebTest 10 Selenium

No discussion, the dataDriven Ant task used with WebTest is simple and powerful!

  • Multi-language support: WebTest 01 Selenium

Selenium RC has bindings in different languages (Java, Ruby, PHP, …) whereas WebTest is bound to Ant which means XML or for instance Groovy with its nice AntBuilder (in fact any of the over 200 languages for the JVM could probably be used). I think that Selenium is missing a real specification language (please don’t talk about Selenese!) like Ant is in this case for WebTest, but I need to give some points to Selenium…

  • Internationalisation support: WebTest 10 Selenium

Using WebTest, you just need to put your language specific strings in property files and use Ant’s built in property task to load the right resources before executing your tests.

Update 05.11.07:
Dan Fabulich correctly indicates that Selenium RC tests written in the same language than the AUT can directly use the application’s i18n resource bundles. Therefore WebTest can’t be seen as winner outside of the Java world.

  • Support for non HTML content: WebTest 10 Selenium

HTML is only one of the content types used by a web application and it’s a common need to have mixed content within the same applications with for instance a PDF file containing the invoice after the checkout. Selenium is limited to HTML content (+XML and text). On the other side WebTest has built in support to work with PDF documents and Excel files as well as Applets and Emails.

Conclusion:

Automated functional tests of web application should become as natural as unit tests. Some tests are better than no tests, no matter which tool is used. Selenium does a good job to introduce newcomers and has many advantages (besides the price) over its commercial model QTP. Nevertheless at least when the size of your suite grows, you should pay attention to your efficiency if you want to last and “the ROI on WebTest is many orders of magnitude higher than any other tool I’ve used” (Lisa Crispin, author of Testing Extreme Programming).

When comparing Selenium and WebTest, 3 categories of web application can be considered. First the applications that are supported both by WebTest and Selenium. This contains most of the applications. Then the ones that use browser features (mostly javascript) that are not yet supported by HtmlUnit. Due to HtmlUnit’s awesome progress in JavaScript support, the size of this category continuously shrinks. The last category concerns applications that use for instance PDF documents or applets and for which Selenium has no support. In all cases where WebTest can be used it is far more efficient than Selenium to ensure the quality of web applications because it provides more feedback and takes less time (both to execute and to analyse results).

When I started writing this post I didn’t expect that Selenium would get such a bad score. Comments are welcome to show which advantages of Selenium over WebTest I’ve missed!

About these ads

100 Comments

  1. October 29, 2007 at 10:59 am

    I’ve never tried Selenium but I’ve worked with WebTest in Grails and found it to be a great tool, and no more than usually difficult to pick up as a web testing beginner.

  2. sandra said,

    October 29, 2007 at 11:03 am

    interesting

    i like the way you compare features and give a final score

  3. Christian said,

    October 29, 2007 at 12:43 pm

    We currently use HtmlUnit and we have a lot of problems with our Tapestry 4.12 application. Even the simplest pages are throwing exceptions, because HtmlUnit can’t parse the Dojo import statements.
    So we have no other choice, as to use Selenium.HtmlUnit doesn’t support enough JavaScript.
    So in my opinion HtmlUnit and WebTest isn’t a really good choice for AJAX applications.

  4. Steve said,

    October 29, 2007 at 2:17 pm

    I’ve never used WebTest but do have plenty of experience of Selenium and concur with your analysis, just maybe not the scores. For me visibility is a huge issue – having a large monitor flashing through the automated tests gives managers a warm fuzzy feeling that makes it easier to justify the amount of time developers spend writing the tests in the first place.

    XPath support & cruise integration wasn’t been a problem since we ran all our tests through firefox and left it for the the QA team to manually walk through the site in IE). Our IE bug rate would probably have been higher but we had a couple of first class frontend guys who made sure everything worked well for all target platforms (Firefox, IE, Opera and Safari).

    Counter that with how dreadfully slow Selenium is on a large suites, abysmal documentation, bugged releases, multi lingual support / RSS support and you’re probably on the money.

  5. October 29, 2007 at 10:01 pm

    [...] Guillemot postou uma comparação entre os dois, e o seu resultado foi vitória do [...]

  6. Jim Priest said,

    October 29, 2007 at 10:53 pm

    Great comparison… I’ve just stumbled into the testing world and so far have been using Selenium and found the Selenium IDE an especially easy way to get into writing tests. I’ve read and experimented a bit with WebTest and found it lacking the same sort of GUI/IDE? Is there something akin to Selenium IDE for WebTest? I want to revisit WebTest because of it’s tight integration with Ant… that really appeals to me but I so far I haven’t found it as user friendly as Selenium.

  7. Rob Di Marco said,

    October 29, 2007 at 10:56 pm

    While I cannot disagree with any of the decisions that you made, I think the score is misleading. The integration with the browser with Selenium is why I use it. It is great for both building the plans as well as debugging things when my automated scripts break. Using the Selenium ant task and Bean Shell for any scripting has been good enough for any project that I have worked on.

    Something that I HATE about Selenium is how it starts up a Jetty server and then fires up the browser to execute against the server. I have been tempted to just build a Selenium client so that when I am just trying to do a simple smoke test, it would run much faster.

  8. Simon said,

    October 29, 2007 at 11:32 pm

    I agree that Selenium is slow, and sometimes fragile, as a result of running a standalone browser. But I think you understate the importance of the realism you’re getting from that. No, it’s not a perfect simulation of a user operating the browser manually, and can’t replace hands-on testing.

    But it’s better than nothing. Tools like WebTest strike me as being somewhat theoretical – they prove that the code works perfectly, but only in an ideal environment that bears little resemblance to production. Real users are using IE or Firefox, and Selenium allows us to do our testing under ‘realistic’ conditions, with flaky browsers that leak memory, and don’t conform to standards.

  9. October 29, 2007 at 11:42 pm

    [...] WebTest vs Selenium: WebTest wins 13 – 5 « Marc Guillemot’s blog (tags: testing selenium webtest comparison java test javascript browser article) This entry was written by delicious and posted on 30 Ottobre 2007 at 01:37 and filed under Del.icio.us. Bookmark the permalink. Follow any comments here with the RSS feed for this post. Post a comment or leave a trackback: Trackback URL. [...]

  10. Ibrahim Salloum said,

    October 30, 2007 at 4:12 am

    We use Selenium in all our projects. But it seems this will end soon.

    Does WebTest support i18n? (i.e. can we test the value of a text field in non-english without forced to write a new ant script)

  11. Marc Guillemot said,

    October 30, 2007 at 8:01 am

    Hi Christian,

    Dojo belongs currently in the category of applications that uses js code that is not yet supported by HtmlUnit (but this may change soon, stay tuned).

    Nevertheless your statement on AJAX application is too general. There are even cases where ONLY HtmlUnit can be used, for instance for load testing AJAX application (see for instance this thread http://www.nabble.com/Load-testing-with-multiple-HtmlUnit-clients-tf2909987.html#a8130458)

  12. Marc Guillemot said,

    October 30, 2007 at 8:21 am

    Hi Steve,

    > having a large monitor flashing through the automated
    > tests gives managers a warm fuzzy feeling that makes
    > it easier to justify the amount of time developers spend
    > writing the tests in the first place.

    how do you justify junit tests with such managers? ;-)

  13. Marc Guillemot said,

    October 30, 2007 at 8:35 am

    Jim,

    capture replay may be tempting in a first time, but this is the “least cost-effective method of automating test cases.”. This sentence is not from me but from Keith Zambelich, co-author of the WinRunner ATS Toolkit.

    WebTest has a recorder too (http://webtestrecorder.canoo.com). A recorder is first very helpful, when you know that recorded scripts should be refactored.

  14. Marc Guillemot said,

    October 30, 2007 at 8:40 am

    Rob, Simon,

    I believe that you miss a very important point:
    – NO tool, absolutely no one, can ensure that your application works “well”. Therefore a tool should help you to catch as much problems as possible. And WebTest gives you information on problems that you will first “see” later in a “real” browser.

    WebTest has nothing theoretical. It is used since years and surely in larger test suites than Selenium.

  15. Marc Guillemot said,

    October 30, 2007 at 8:42 am

    Ibrahim,

    i18n works just fine. If I correctly remember there is even an old Japanese translation somewhere.

  16. Steve said,

    October 30, 2007 at 9:14 am

    >> having a large monitor flashing through the automated
    >> tests gives managers a warm fuzzy feeling that makes
    >> it easier to justify the amount of time developers spend
    >> writing the tests in the first place.

    >how do you justify junit tests with such managers?
    It’s never come up. I’m aware of a couple of organisations that don’t believe in unit tests, but you wouldn’t catch me working there!

  17. October 30, 2007 at 1:02 pm

    [...] the contest is Marc Guillemot, an admittedly biased WebTest committer. Here are his scores and explanations about this [...]

  18. testerbcn said,

    October 30, 2007 at 2:16 pm

    I am a selenium remote control user. I am not really convinced about your points. I really like using selenium, I can do almost everything without to many problems. It had a real short learning curve.
    – Reports. I use log4net in selenium rc, and i can easily find errors.
    – Speed. It’s doing what a real user does. So running your tests and looking at it can help you detect other errors not contemplated in your test.
    – Extensibility: You can write a lot of extra code in C#, Java, …
    – Analists can use selenium ide, very easy to use.
    -…

  19. SK said,

    October 30, 2007 at 7:03 pm

    I think WebTest is quite cool. I just wished that you had support for http://www.cubictest.org so you could create your tests there and export them to WebTests. Or maybe even run them from there.

    Well, I guess I’m quite biased.

  20. Simon said,

    October 30, 2007 at 7:43 pm

    Marc –

    I don ‘t use ‘theoretical’ in a perjorative sense. But no client will ever be using the application on the particular engine that WebTest uses, meaning that while it’s nice to know it works on something, it doesn’t really matter. In contrast, our Selenium tests run on Firefox, and they run on IE, and they catch a lot of the problems that occur when someone hasn’t written something in a cross-platform manner. Can WebTest do that?

    Note – I know that no tool is perfect. But for this kind of test, it’s my view that the closer the test is to production reality, the more useful it is.

  21. Marc Guillemot said,

    October 31, 2007 at 8:01 am

    > – Speed. It’s doing what a real user does.
    > So running your tests and looking at it
    > can help you detect other errors not contemplated in your test.

    I’ve already heard funny things about test automation but never this one. Personally I think that the tests are never fast enough.

  22. Marc Guillemot said,

    October 31, 2007 at 8:04 am

    Hi SK,

    I didn’t know cubictest. I guess that it wouldn’t be specially complicated to build a plugin for WebTest. Perhaps should we discuss this directly?
    Nevertheless I’m a bit sceptical: such a generic strategy normally tends to reduce the tool to the least common denominator of the possible targets.

  23. Marc Guillemot said,

    October 31, 2007 at 8:13 am

    Simon,

    I guess that I have to write a future post on this subject.

    Saying that it’s better to run tests in a real browser, because it’s where the application is really used, is a too short statement. What is required is to have a fault model: what kind of errors do you wish to catch? Which tool is able to catch them? WebTest is able to catch many problems than other tools can’t catch or first see later.

    Additionally, yes, HtmlUnit is able to simulate (with the limitations already mentioned in my post) different browsers (currently FF and IE) and to catch errors due to scripts that haven’t been “written in a cross-platform manner”.

  24. testerbcn said,

    October 31, 2007 at 9:59 am

    Is it so funny? I agree with Steve about visability, and with Simon about the hands-on testing. Ok, maybe speed is not the right word, but it surely has some advantages. With selenium you can see what is happening, how a page is working in firefox and iexplorer, etc. and it is mostly similar to hands-on testing…

  25. Camille ALBERT said,

    October 31, 2007 at 10:15 am

    Bonjour Marc,

    Very good job, i will see now how i can use WebTest on our project.

    In my opinion html tests (and more general tests of the presentation layer) are too expensive and less usefull than the other layer tests (logical/process layer, business layer and persistance layer). So on our project html tests are not done, developers test on their computers thanks to a light version of the application (done with Jetty for the test). It’s not automated, but … maybe with your tool…

    When i worked on our “test-container” i used MockEJB, Jetty, commons-dbcp and other goodies to test our persistance layer, our business layer and our logical/process layer (with Web services calls in the tests, that’s the reason why we use Jetty) it was’nt too difficut to use. Is WebTest easy to use ?

    I agree with you when you say that for the automated tests the browser is useless.

    Bon courage pour la suite.

    Camille.

  26. Sebastien Brunot said,

    October 31, 2007 at 11:00 am

    Hi,

    we’re testing real life applications with Selenium for more than one year now. In real life, you’ve got bugs in IE that does not occurs in Firefox, etc… Running the tests in the real browser IS a mandatory feature for us.

  27. October 31, 2007 at 11:57 am

    I second the statements of Sebastien and Simon. While I agree with some of your points (speed, extensibility), Selenium’s fundemental advantage is is the real-life behaviour. Webtest-Tests could run perfectly, if CEO’s and clients’ Firefoxes would choke it would be a disaster. Concerning the documentation: after installing SeleniumIDE, it will show you the documentation for every single command while choosing it.
    And do not underestimate the addictive nature of Capture-n-Replay here. IMHO it is better to cover some usecases with a simple Capture-n-Replay tool than none because of time constraints.

  28. Steve said,

    October 31, 2007 at 5:43 pm

    > In my opinion html tests (and more general tests of the presentation layer) are too expensive and less usefull than the other layer tests (logical/process layer, business layer and persistance layer).

    The point of automated web tests is that they exercise the whole application (including databases, etc), rather than just a single layer. They are a hefty investment and you won’t see the payoff immediately, but they’re absolutely essential for maintaining productivity.

  29. Camille ALBERT said,

    October 31, 2007 at 6:09 pm

    > The point of automated web tests is that they exercise the whole application (including databases, etc), rather than just a single layer. They are a hefty investment and you won’t see the payoff immediately, but they’re absolutely essential for maintaining productivity.

    I agree with you, but those tests you describe are my tests of the logical/process layer (with web services calls to the business layer in the test), and you’re right they are really important.

  30. Kent Tong said,

    November 1, 2007 at 3:08 am

    Possible to develop a middle layer that allows one to write one set of tests that can be run on both WebTest and Selenium? This way people can get the best of both worlds by frequently runing WebTest and running Selenium nightly. It should automatically skip tests that obviously don’t run in one environment (possibly with the help of an annotation if it is required).

  31. Mike said,

    November 2, 2007 at 4:49 pm

    Integration with the development environment is available through the Selenium IDE. You can save your test as Java code and run it during the integration phase in Maven. The test does not need to run through the browser.

  32. Shimtest said,

    November 3, 2007 at 12:26 am

    A useful criteria would be speed of test case creation of Selenium vs. WebTest. I’ve just fooled around with WebTest for part of a day but I think it may be much slower. We’re trying to push automation up to feature test timeframe, I think Selenium is more realistic for that scenario.

  33. Dan Fabulich said,

    November 3, 2007 at 1:43 am

    Many of your comments are quite fair (especially remarks on speed/scalability and the point about capturing JS errors). As so many of the other commenters have said, I think there’s a place for Selenium and a place for WebTest.

    The metaphor I like to use is that Selenium (the chemical element) is an essential nutrient, but it is toxic if taken in excessive doses. Much the same is true of Selenium UI tests. :-)

    UI tests can’t and shouldn’t replace unit tests. Selenium tests are integration tests with the browser as a platform; HtmlUnit/WebTest tests mock out the browser platform, focusing on the webapp as a unit. You should test your code at the right level; that usually means you’ll need to write some of both.

    I think your remarks are unfair when it comes to discussing reports, data driven testing and internationalization support. Selenium RC is a browser automation API; it’s not a framework for managing automated tests. It has a command to start a browser and a command to click on a button; it doesn’t have a command to start a test or fail a test.

    When writing tests that use Selenium RC, you normally define your tests in a test framework like JUnit, TestNG, PushToTest, NUnit, pytest, phpunit, Perl’s Test::More, etc. Some of those frameworks are really only suitable for simple unit tests; others have powerful reporting libraries. Some of them support defining tests using data (e.g. TestNG) and others don’t (JUnit 3). Some of them are feature limited out of the box, but have fantastic open-source extensions available on the web. It depends on what language you’re using and what framework you’re using underneath.

    It sounds like you were using JUnit 3 for a lot of your tests; I think you may prefer to write Selenium tests using TestNG in the future [if you ever do use Selenium in the future ;-)].

    For internationalization support in particular, I think Selenium RC is the clear winner over WebTest, simply because Selenium RC tests can be written in the exact same language as the AUT. That means that you can use your native language’s i18n resource bundles directly in your tests; you don’t have to repackage them in a form WebTest can understand. (If you’re lucky enough to already be working in Java using i18n bundles WebTest understands, you could call this one a draw.)

  34. November 5, 2007 at 12:26 am

    Some people are wondering how Canoo WebTest or Selenium compare to Windmill or WebDriver

  35. Robert Zimmermann said,

    November 5, 2007 at 10:06 am

    Reports: If running Selenium in HTMLRunner Mode (also possible with selenium remote-control) the result will be a nice html file containing the entire testsuite run (colored in green and red as any unittest framework does).
    For the x-unit mode of selenium remote-control (at least for junit) I am currently working on an similar html-based reporting support. So in case of an assertion there is not only an junit traceback but also a bit history what happended before.
    There is also a brilliant screenshot mechanism built in selenium remote-control helping to see what went wrong especialy in an unattended nightly automated run.

    Browser fidelity: I think if manual testers would use an automated part driven by selenium they could, and that every day and/or every new release, focus on display faults or new testcases just by watching selenium doing the tedious part of their work.

  36. Marc Guillemot said,

    November 5, 2007 at 10:31 am

    Hi Dan,

    I’m pleased to see comments of a famous Selenium committer here.

    Concerning your “contestations”:

    Internationalization support
    You’re right. Outside of the Java world Selenium RC tests written in the same language as the AUT have the advantage of being able to reuse i18n resource bundles. I’ll update my blog post to add this information.

    Data driven:
    You can use TestNG for instance when you write your WebTest tests as Groovy code. Therefore this can’t be put as disadvantage of WebTest and the simplicity of dataDriven remains.

    Reports:
    It seems that I have to explain a bit more what makes WebTest’s reports so particular and why no framework can provide this kind of reports if Selenium doesn’t provide them. Besides details of all actions performed during the test, the report contains all responses received from the server (as well as dump of dynamically modified pages). This is a huge advantage: you don’t need to debug the tests, just to look at the reports for instance to find that a test fails due to some problem that occurred a few steps before the one that actually fails.
    So, yes this is an advantage of WebTest. A WebTest users even once wrote concerning the motivation to use WebTest: “the reports, forget all other reasons”.

    Finally I don’t agree on the distinction you (and others in these comments) make between real and simulated browsers. No test tool (absolutely none, it’s not the question whether Selenium or WebTest) can ensure you that an application works correctly. Sorry if I disappoint you here. Test tools can only ensure that different kinds of errors don’t happen. Selenium will help to find some errors that WebTest can’t (currently) find but WebTest can find more errors and can find them earlier.

  37. Marc Guillemot said,

    November 5, 2007 at 10:48 am

    Hi Andre,

    for me WebDriver is a bit like using HtmlUnit directly: you have the whole power… and the whole responsibility. Compared to WebTest, you won’t have very important things like the reports. WebDriver aims to support many browsers, such approach often leads to be the less common denominator between the supported targets.

    After 30 secondes surfing on Windmill’s site: I’m missing a text representation of the tests. The site doesn’t show at all test automation as a software engineering activity. I don’t see how to build maintainable test suites with what they call an IDE (here gain! ;-)).

  38. Dan Fabulich said,

    November 5, 2007 at 3:32 pm

    Hi Mark,

    Your comments about my “fame” flatter me. ;-)

    I never meant to imply that “data driven” was a disadvantage of WebTest, but just that Selenium RC tests using TestNG are easily data-driven. “Data driven” is a “tie” in my opinion.

    Reports: That’s an interesting feature of WebTest with which I wasn’t familiar. Is there an example of this on the web somewhere I could look at? I see the image on the homepage, but it doesn’t highlight the detail you describe. With that said, it sounds like WebTest is a clear winner on this point.

    [FYI, the data is all available in Selenium RC, but it's available as special commands, not as a nice report. For example, getHtmlSource will spit out the (dynamically modified) HTML, but you have to explicitly run it; we don't record the HTML on every page load. Similarly, as Robert pointed out, you can use the captureScreenshot command to record PNG screenshots of the browser, but you have to explicitly call that command yourself. HTTP requests/responses are recorded in RC logs, but at the debug level. You can see them (and way too much other data as well) with the -debug argument, or you can configure your JDK log parameters to record the HTTP stuff separately.]

    Real/Simulation: Nobody would argue that using a real browser will find *all* bugs, but I don’t think you’re right that WebTest finds “more errors” than Selenium. I think you’re right that WebTest can find certain errors *earlier*, partly by being faster, and partly by handling JS compile errors directly instead of waiting for them to cause a test failure.

    (IMO JS compile errors should be caught using other command line tools, e.g. minifiers like the Dojo compressor… you shouldn’t even have to simulate the browser to catch those.)

    But Selenium does catch bugs you won’t catch in a simulated browser. Some browsers are simply non-standard in their behavior; it’s unrealistic to expect HtmlUnit/WebTest to simulate all of their quirks, especially when you’re doing fancy DHTML tricks.

  39. Marc Guillemot said,

    November 5, 2007 at 3:59 pm

    Hi Dan,

    for me you’re the most famous Selenium committer: you’re the only one who wrote that HtmlUnit js support is so good! ;-)

    You can look at WebTest’s selftests to see how reports look like: http://webtest.canoo.com/webtest/reports/selftest/WebTestReport.html
    Due to the facts that these tests are done to test WebTest itself, the tests are very short and the html page are a bit special: they contain only the stuff needed to test a particular WebTest feature. The consequence is that the report features are not as useful there as in normal projects.

    The Dojo compressor will only find JS compile errors, not runtime errors. I agree that ideally a project that heavily relies on client side JS should have JS unit tests and should find the JS errors before a single browser test is started.

    IMHO Selenium could provide far more information if it uses everything what Firefox provides. For other browsers… (probably still possible to catch JS errors)

    I agree that HtmlUnit will never simulate all quirks of real browsers. Just the ones used by “popular” AJAX libraries seems a reachable goal and will cover most of usage.

  40. Dan Fabulich said,

    November 7, 2007 at 6:16 am

    In my experience, JS runtime errors will cause runtime failures when tested; you’ll catch them just as early with Selenium as with WebTest.

    Thanks for the link to that report… We’ve got something to work towards. ;-)

  41. Marc Guillemot said,

    November 8, 2007 at 9:06 am

    I disagree: as far as I could test it, Selenium doesn’t see any JS runtime error. It only see the consequence of it if this is covered by your test. As example, if an onclick handler failed to update some part of the page due to a runtime JS error, Selenium won’t notice it. Your test will first “see” it if it tests whether the page has been updated appropriately. I guess that Selenium could catch this kind of errors by registering on onerror handler.

  42. 001Dark said,

    November 26, 2007 at 9:24 pm

    hi i’m from russia :)
    u blog cool useful, may post u inf. on my site ?

  43. November 27, 2007 at 9:04 am

    [...] WebTest vs Selenium: WebTest wins 13 – 5 « Marc Guillemot’s blog: בדיקת אפליקציות ווב: השוואה מקיפה בין סלניום למבחןרשת. הזוכה המאושר: מבחןרשת, התוצאה: 13-5 [...]

  44. December 4, 2007 at 10:45 am

    [...] the discussion in Grails-user mailing list following my previous post “WebTest vs Selenium“, Marc Palmer and James Page requested the creation of a kind of meta DSL in Groovy for [...]

  45. ftorres said,

    December 4, 2007 at 12:52 pm

    For those interesting in writing web testing in C#, VB.NET or IronPython using the different flavors of Visual Studio, see http://www.InCisif.net.

  46. December 12, 2007 at 7:58 am

    I’d prefer reading in my native language, because my knowledge of your languange is no so well. But it was interesting!

  47. December 17, 2007 at 12:38 am

    [...] из разработчиков HtmlUnit и Canoo WebTest набрел на интересный анализ — сравнение двух тестовых инструментов Selenium и [...]

  48. rajeena said,

    December 27, 2007 at 7:03 am

    I have used selenium for record and play back only. but i want to test application using java script. from where i will get proper guidelines for the same

  49. Denis said,

    January 3, 2008 at 3:49 pm

    Marc,
    What do you think about Selenium with Integrated Extractor. Look here: http://www.kerrybuckley.com/2006/11/16/integrating-selenium-with-exactor/.

  50. Nate Kirby said,

    January 6, 2008 at 12:51 pm

    Can you write DSLs in WebTest?

  51. Marc Guillemot said,

    January 8, 2008 at 8:46 am

    Denis,
    I don’t know Integrated Extractor, but it seems to me that it’s just a way to fill a Selenium test wit data. WebTest’s dataDriven is really easier to use ;-).

    Nate, look at my recent post:

    http://mguillem.wordpress.com/2007/12/04/webtest-as-universal-dsl-for-automated-web-testing-in-groovy-thanks-to-its-ant-roots/

  52. Al said,

    February 1, 2008 at 12:14 pm

    What about Watir?

  53. Setya said,

    February 6, 2008 at 6:40 am

    Is WebTest suitable for testing RIA based applications such as RAP ?

  54. Marc Guillemot said,

    February 6, 2008 at 9:20 am

    What matters is what is used on the client side. WebTest can be used to test Ajax app (with the limitations already mentioned) or applets but not Flash applications.

  55. ahndunk said,

    February 18, 2008 at 10:46 am

    Interesting post about selenium.
    for Setya, WebTest is not suitable for testing RIA based applications.

  56. February 26, 2008 at 12:04 pm

    [...] Guillemot postou uma comparação entre os dois, e o seu resultado foi vitória do [...]

  57. Chris Jones said,

    February 28, 2008 at 12:04 am

    I’ve used both frameworks for automated testing heavily (httpunit/htmlunit for 3 years and selenium w/rc + junit for 2 years now). For me, Selenium is the clear winner for the following reasons:
    1. Watching selenium do its thing in a real browser is a huge help. When I’m writing a test for our feature-heavy gwt + jsp application, I can step through and watch what is happening. This always helps me identify and solve problems sooner. When I used a faceless tool (httpunit/htmlunit), I simply couldn’t do that and it heavily limited my tests.
    2. Selenium handles anything I want to do in the browser, there are no limits there (except when it come to file uploads, but there is a way around that). Gwt, dojo, and tons of other toolkits out there are going to push the limits of the browser and that’s what I’m coding for, this is the web 2.0 era after all. I found that with the simulated faceless browser approach, you were always going to run into something you couldn’t test and then you’re either stuck behind the times or stuck with testless code.

    Selenium is definitely slow, but to me its worth it for the problems I want it to solve. In terms of the argument that webtest can find bugs earlier and faster, I believe that straight-up unit tests will get you there much faster than either webtest or selenium. Gwt also has a facility for unit testing ui code that is faster than either of these tools.
    The niche that Selenium and WebTest fill, however, is that of an end to end test with the intent of getting as close to actual user experience as possible. You have to be strategic about how much testing to do at this level, but solid coverage of your basic paths is well worth the effort. For this niche, I found Selenium to be the winner.

  58. sachin Kumar said,

    May 9, 2008 at 12:17 pm

    In web testing i am facing a problem….

    During the execution of the selenium script it is not identifying text box and giving the statement with red mark. So, if suppose i want to skip that step and continue with next steps execution as normal what should i do?.. By right clicking and selecting “Execute this command x” command just it is executing one statement. So, Manually i need to execute all statements in the script.(Which is waste of using automation). So, anybody can help me for my issue. Please ans. me as soon as possible

  59. Pathy said,

    May 20, 2008 at 5:34 pm

    Any one did a comparison between QTP and Selenium? If yes, please share with us.

  60. June 12, 2008 at 2:51 am

    Hey Marc,
    I agreed totally — based on my past experiences where I have implemented automation framework with both tools; believe WebTest wins the race for sure.

    You compared very well; I appreciate for this. I just want to say Reporting and Maintainability of WebTest is far more ahead then of Selenium.

    Thanks,
    Nimesh.

  61. Peter said,

    June 17, 2008 at 6:31 pm

    I think this post is great, the person who wrote it recommends selenium ..

    http://java4you.wordpress.com/

  62. Marc Guillemot said,

    June 18, 2008 at 6:59 am

    For me someone that uses the term IDE for a recorder without any comment gives indications on his experience with test automation. I would never call WebTestRecorder an IDE (even if this poster didn’t see some features that makes this recorder a bit less stupid than Selenium’s one).

  63. qqrq said,

    June 19, 2008 at 2:33 pm

    Hello

    Well, you write, that Selenium has bad documentation. If they’d got rid of broken links it would’nt be so bad.
    Also you have to keep in mind, that not everyone likes XML. :) I get a headache each time I look at WebTest’s.

  64. Marc Guillemot said,

    June 19, 2008 at 2:40 pm

    You’re free to write your tests for instance as plain Groovy code when you don’t like XML (I guess that any language on the JVM that is able to work correctly with Ant could be used as well like JRuby).

  65. qqrq said,

    June 20, 2008 at 5:53 am

    OK, that’s cool.

    The main argument, I choose Selenium for, is, as you call it, “browser fidelity” – it runs in a browser and gives us “real” situation. I think this is main advantage for front-end developers.

    BTW Sorry for my english – syntax is much complicated than one in JS. ;o)

  66. Victor Klimov said,

    June 20, 2008 at 9:03 am

    Real ‘Test automation':

    It seems surprising that I was not able to find a ‘real’ automated test engine on the net, some kind of bot or crawler for testing robustness of web applications.

    Such tool would be something like the good old perl Mechanize module with analytic abilities and test data generation:
    start with a page, go recusively into all odd corners, click on all buttons,
    fill out forms with all kind of impossible random data and submit and
    in each possible way try to trigger exceptions.

    Or didn’t I look where I should?

  67. Victor Klimov said,

    June 23, 2008 at 3:40 pm

    One word: fuzzing

  68. michaelc said,

    June 25, 2008 at 9:11 pm

    it might be me, bad timing .
    but in the 4 times I have tried to get to the webTest products the server has but been available.

    I think there in need to some Automated tests on server availability.
    I would like to run it thru the steps, is there a mirror some where ? \

    Network Error (tcp_error)

    A communication error occurred: “”
    The Web Server may be down, too busy, or experiencing other problems preventing it from responding to requests. You may wish to try again at a later time.

    For assistance, contact your network support team.

  69. rr said,

    July 4, 2008 at 5:07 pm

    Hi,

    Canoo webtest works with aplication based in .net? . example C#.

    test my aplication, this present error HTTP 404.

  70. great said,

    July 7, 2008 at 3:38 pm

    hi

  71. Marc Guillemot said,

    July 8, 2008 at 8:27 am

    @rr
    when you look at this page, does your browser care if the website is programmed in php or with something else? No, and this is exactly the same with WebTest.

    @michaelc
    WebTest’s website looks ok for me. Probably bad timing.

  72. rr said,

    July 10, 2008 at 2:08 pm

    Hi,

    I have a problem with a aplication, don´t redirect , entered user and password, but this redirect go to page principal… i tried with and only .

    Content of current window changed, it will become current response
    Current form set to none
    Current response now

  73. rr said,

    July 10, 2008 at 2:09 pm

    config and invoke

  74. Vishal_Pai said,

    July 17, 2008 at 5:48 am

    Bonjour Marc,
    I came across your blog while evaluating open source tools for my testing project.
    I agree WebTest has a more maintainable framework than Selenium (and with WebTest’s enhanced documentation it sure helps!).
    But I checked out Selenium (as a bundle of IDE and RC together)a few points where I think Selenium scores on WebTest is
    1.Functional testers without knowledge of scripting could really work around the Selenium IDE plugin in FF and record long scripts.
    2.The browser based replay does end the mysticism associated with WebTest results.WYSIWYG replay is a big confidence booster for harrowed testing people I guess.

    Nevertheless WebTest comes out a winner when you consider it’s documentation,speed ,reports and datadriven capabilities.Also lot of people i spoke to were comfortable with Groovy and XML than using Selenese.

    In a ideal situation it would have been wonderful to have WebTest with an IDE option for Firefox to record and also Replay with datadriven capabilities while the recorder could have been used for getting scripts into the WebTest framework where coverage and script errors could have been taken care of.

    At the end of it I have chosen WebTest as the primary tool,while allowing functional analysts and testers to play around the application in Selenium IDE.

    Works good for me this way.

    Vishal

  75. Marc Guillemot said,

    July 17, 2008 at 7:37 am

    Vishal,

    thanks for the kind comment.

    To (1): this is a marketing argument normally used by tools vendors but that is totally wrong. Without technical knowledge, you won’t be able to write good, maintainable tests, no matter what the tool is.

    I would never use the term IDE for a recorder, but WebTest has a recorder as well and… it will be able to drive “visual” browsers in the future.

  76. July 24, 2008 at 8:51 pm

    [...] found on his blog a biased comparison of canoo and selenium, that essentially backs up the points I was trying to make.  It seems one of [...]

  77. August 13, 2008 at 2:39 pm

    [...] This blog post shows an excellent comparison between the two tools. It favors WebTest due to some of its greater feature set. This post however favors Selenium due to the ease of use. [...]

  78. John said,

    August 14, 2008 at 9:10 pm

    Selenium is a great tool and a brilliant idea, just the “record and reply” style usually cannot be used for practical testing. There are something missing there i.e, there should be another tier over Selenium to make tests easier to write and maintain. We created the Tellurium automated testing framework for this purpose. It is UI module centric and very expressive. You can use DSL to write tests. It also provides data driven testing feature. The project is at:

    http://code.google.com/p/aost/

    Thanks.

  79. jim said,

    August 25, 2008 at 10:21 pm

    Hi i have a question im tried to select a functional testing tool open source hopefully but i have it some problems with the WebTest tool, the problem is when i do it a test in a site using jboss seam (Faces or Ajax) they dont recognize the lateral menu that i using for my app. Someone have it the solution of that problem. Thanks for your help ans if you now other functional testing tool i appreciated.

  80. Mhia said,

    October 30, 2008 at 4:05 am

    Hi there,

    Im Mhia and new in testing and it was my first time to use web test. DO you have any link or forum ?

    Thanks in advance!

  81. rex said,

    November 5, 2008 at 9:14 pm

    Does WebTest support GWT type code?

    • August 31, 2009 at 2:18 pm

      @rex:
      You don’t need neither Selenium nor WebTest if you are using GWT because GWT has its own Unit Testing environment based on JUnit.

  82. Mark Levison said,

    February 25, 2009 at 5:45 pm

    This an interesting comparison but I think far more interesting would Selenium Core vs. WebTest. Any thoughts on how the score would change?

    I ask because as attractive as WebTest looks to me, my client will probably use Selenium Core.

  83. Zahid said,

    March 4, 2009 at 7:06 pm

    Its nice to read this article. But it may be bit dated.
    eg you can verify Java script error with selenium in say “IE” – by
    turning on Tools-InternetOption->Advance -> “Display notification for script error” .
    Does webtest provide any good “recording” tool ? If not – then most people
    will not use it. Also since the output language is in lots of language(html,.net,java etc) – the use will just increase.

    Thanks,
    Zahid

    http://www.jroller.com/selenium

  84. Marc Guillemot said,

    March 5, 2009 at 8:13 am

    Zahid,

    if I correctly understand your strategy to capture JS errors with Selenium and IE, you just wait that the tests get blocked because the error notification popup is shown? Is it correct? In this case, this is not at all what I understand with test automation.

  85. krishna said,

    March 11, 2009 at 7:16 am

    The blog posts back to October 29, 2007 .. It is been 2 yrs and lot of changes would have been made in both the Tools .Can any one tell the latest comparision . I have seen selenium IDE which is good for recording , but managing part is still not clear ..

  86. December 17, 2009 at 3:23 pm

    мне кажется: прелестно.

  87. ecards said,

    January 31, 2010 at 10:49 pm

    1) The article and comments are an outstanding first education for someone new to testing tools (but not to development).

    1) The link in the comments to show a sample report is broken, I’m sure I can search for an example but to let you know it doesn’t work for people reading here:

    http://webtest.canoo.com/webtest/reports/selftest/WebTestReport.html

    3) The one argument against WebTest that seems to never be refuted is that Selenium catches more real-world errors. It seems the response to this is something like “no tool catches all errors”, which is not really a refutation.

    Btw, I don’t know yet the answer for (3), it seems also the people making the claim for Selenium are basing this off of intuition rather than specific research about what % of errors are caught by each approach.

  88. Andrew said,

    February 11, 2010 at 2:39 pm

    Hi!

    We have just tried using WebTest for our application. The problem is that the support for JavaScript is not good enough yet (we are using Prototype, Flotr and several other libraries). I like the idea of automated tests that need no display, but right now we will have to deploy Selenium – because it behaves exactly as Firefox would.

    While WebTest is ideal for testing web apps with limited JavaScript, it fails on JS rich web apps. So for us the scoring is very misleading – it’s more like 0:1 (Selenium wins hands down).

    But I’m sure the effect will be less obvious in the future as HtmlUnit becomes more and more JS capable.

  89. Andrew said,

    February 12, 2010 at 10:33 am

    Quick update: we have solved the Prototype issues (added after AJAX calls… hmmmm), but Flotr remains a problem. However, testing Flotr is not a high priority for us, so we’ll still try to use WebTest.
    Still, browser similarity is a very important point when comparing solutions, so we might end up with Selenium if we encounter more unsurmountable problems. Performance matters, but only if the product works first. :)

  90. Vijay Pillai said,

    March 26, 2010 at 6:14 pm

    Interesting !!
    We do almost all of our automation in Selenium and I did not come across a comparison like this when I made a decision to choose Selenium as the tool we want to stick to.

    But there is another interesting feature provided Selenium in the form of grid. This help a tester run hos tests in parallel and hence complete say 2 tests (FF 3.5 & FF 3.6) in the time required to complete one. Also, using Selenium grid, I now have a framework which help me control what machines I want to have my tests running. I may not want to run it on all browsers all the time.

    Is it possible to do something like this using WebTest??

  91. June 7, 2010 at 5:01 am

    I have started to use Selenium in our project. The “convert recorded” steps to c# is really great. It allows us to create helper classed that we use as building blocks for future tests. Our library of helper methods is so rich now that we often times don’t even have to record to build a new test, but instead we just call combinations of our helpers.
    I have also created a fluent API to help make it easer to create xpath expressions. Check it out if you are interested:

    http://www.unit-testing.net/CurrentArticle/How-To-Make-Web-Tests-Less-Fragile.html

  92. December 23, 2010 at 10:08 am

    [...] couple of weeks ago, Marc Guillemot published a blog post comparing Canoo’s WebTest and [...]

  93. December 31, 2011 at 1:21 pm

    You can certainly see your expertise in the work you write. The sector hopes for more passionate writers like you who aren’t afraid to mention how they believe. At all times follow your heart.

  94. June 29, 2012 at 6:33 am

    [...] I’m not going to cover every aspect of the differences in detail; for that refer to this excellent blog post comparing the two. Depending on your application and your exposure, I hope the results cited there will make it [...]

  95. wedding veil said,

    October 14, 2012 at 4:43 pm

    Generally I do not learn post on blogs, but I wish to say that this write-up very compelled me to take
    a look at and do so! Your writing taste has been amazed me.
    Thank you, quite nice article.

  96. Lincoln said,

    November 23, 2012 at 9:41 am

    Heya! I understand this is sort of off-topic but I had to ask.

    Does building a well-established blog such as yours require a lot of work?
    I am completely new to writing a blog but I do write in my diary on a daily basis.
    I’d like to start a blog so I can share my experience and thoughts online. Please let me know if you have any recommendations or tips for new aspiring blog owners. Thankyou!

  97. Rajeshkannan said,

    March 14, 2014 at 10:34 am

    Hi,
    Im trying to setup canoo but am getting the below exception. COuld you please help me in resolving it
    Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/tools/ant/
    launch/Launcher
    Caused by: java.lang.ClassNotFoundException: org.apache.tools.ant.launch.Launche
    r
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    Could not find the main class: org.apache.tools.ant.launch.Launcher. Program wi
    ll exit.

  98. Rajeshkannan said,

    March 24, 2014 at 5:27 am

    Hi,

    Im using webtest for my webapplication functional testing. its failing in one step where i try to click a link i.e href


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: