WebDriver: capture JS errors while running tests

October 11, 2011 at 9:07 am (WebDriver)

The stack trace of an exception in a log file is a sign that something went wrong. Even if nobody complains it is wise to take a bit time and investigate it before it really hurts.

Exactly the same applies to JavaScript errors in a web application. I believe that each JavaScript error should be considered as a bug and they should all be fixed before delivering, no matter if they directly impact the user experience or not.

Many great tools (JavaScript Console, Firebug, WebDeveloper Toolbar, …) allow to see the JavaScript errors while developing with Firefox as well as with other browsers and a wisely used error handler can report JavaScript errors to the server in production.

Sadly the support is not good while running automated tests. HtmlUnit provides a great help for that (see for instance this) but often integration tests should run in “real” browsers and not in HtmlUnit (for some good and many bad reasons). I don’t know currently any browser automation tool providing access to the JavaScript errors and to other information available in browsers like Firefox or Chrome that can be helpful for developers conscientious of the quality of their work

WebDriver is here not better than its concurrents and the issue “API for checking for JavaScript errors on the page” is opened since 2 1/2 years without any sign of life from the project’s members. The Selenium 1 issues SEL-522 and SEL-613 are equally asleep. Luckily WebDriver provides the possibility to configure additional extensions when using the FirefoxDriver, what allows to improve the situation.

Tester’s friendly application and injected error handler

The simpliest way to catch JavaScript errors for later retrieval is to add something like that

<script type="text/javascript">
window.jsErrors = [];
window.onerror = function(errorMessage) {
  window.jsErrors[window.jsErrors.length] = errorMessage;
}
</script>

in every page of your application (see for instance this post).

This allows you to retrieve the captured errors with

((JavascriptExecutor) driver).executeScript("return window.jsErrors");

This is a simple solution but it has different drawbacks. First you need to add this code at the beginning of all your HTML pages. If you’re in the ideal situation to be both developer and tester of the application it’s “just” one Ctrl+C and a lot of Ctrl+V. If you need to convince your colleagues or your management that the web pages should be modified for testability, it will be more difficult. From my experience it is not always easy to make such a requirement accepted.
To overcome this problem we could imagine injecting the handler while running the test:

String script = "window.collectedErrors = [];"
  + "window.onerror = function(errorMessage) { "
  + "window.collectedErrors[window.collectedErrors.length] = errorMessage;"
  + "}";
((JavascriptExecutor) driver).executeScript(script);

but besides the fact that it has to be done on every page, we can’t do it early enough: a lot of JavaScript may already have been executed in the page when our executeScript statement is reached.

The second problem of an error handler per window is the difficult access to the captured errors. The JavaScript errors won’t necessarily occur in the currently focused window forcing you to iterate through all (i)frames and windows to check for errors. This can still be seen as a minor issue but the real problem is that captured errors get lost when a new document is loaded. At the end this solution allows to capture some JavaScript errors but not all JavaScript errors.

Add support within the FirefoxDriver

Firefox allows extensions to get notified of JavaScript errors, no matter where they occur.
This mechanism is used for instance by Chris Pederick’s famous Web Developer Toolbar to display a warn/error icon when an error has occurred on a page. Once you’ve succeeded in overcoming the sandbox mechanism separating extensions and loaded pages, it is quite easy to write a new Firefox extension that captures the JavaScript errors and makes them available in each content window.
WebDriver on its side allows to add custom extensions to run in a FirefoxDriver. If you simply add this errors capturing extension to the FirefoxProfile you’ll have access to all JavaScript errors from within your tests. Luckily you don’t need to write it again and you can simply use my JSErrorCollector (Apache 2 Licence). It allows to write following:

...
import net.jsourcerer.webdriver.jserrorcollector.JavaScriptError;
...
final FirefoxProfile profile = new FirefoxProfile();
JavaScriptError.addExtension(profile);
final WebDriver driver = new FirefoxDriver(profile);
// ...
// navigate to some pages
// ...
List jsErrors = JavaScriptError.readErrors(driver);
assertThat(jsErrors, is(empty())

The JavaScriptError class holds the message, source name and line number of a JavaScript error just like what you can see in Firefox’s JavaScript console.

Next steps

Other browsers

It would be nice to have a simple access to JavaScript errors as well in the other browsers supported by WebDriver. For HtmlUnit it is a no-brainer. For Chrome it should be quite simple as an extension can register a global error listener. No idea what concerns IE and Opera.

Integration

Ideally the API should be provided natively by WebDriver, let’s see what will be the interest in my JSErrorCollector (follow issue 148 to stay informed).
Retrieving the JavaScript errors is a first step on which different interesting features could be based. Verifying the absence of JavaScript errors after each test (using for instance JUnit’s @After) works fine but is tedious. One interesting feature would be to let the tests directly fail on the first JavaScript error (I’ll address wrapping drivers in a future post). Alternatively a precise dedicated reporting would have great benefits too. Some filtering features to ignore errors in third party libraries would be a nice feature as well.

JSErrorCollector is open source therefore patches are welcome. If you can’t/don’t want to contribute directly, I’m available to work on it on a contract base.