Why Google should better use HtmlUnit to avoid JS error on its start page

Like many other testing tools, WebTest uses Google‘s website for simple code examples. It has the advantage to be a well known website, pretty fast and with simple code.

http://www.google.com used to be a simple page

For some time WebTest users started to report that even simple examples from the demo tests available when creating a new project failed due to JS errors like this:

As this test used to work, this means that Google’s start page isn’t so simple any more that it used to be. It was predictable that this page would change one day but it is quite embarrassing as we use this kind of tests to show how easy and powerful WebTest is even for sites using a lot of JS stuff.

http://www.google.com has really a JS error!

HtmlUnit‘s JS support is quite good (and improved continuously), but of course not (yet 😉 ) as good as the one of “real” browsers, therefore I’ve started to search for the cause of the problem in HtmlUnit, looking for missing/incorrect implementation of some JS functionalities. At the end I’ve found that the problem occurred only when setting directly the value attribute of the field rather than calling the type(...) method on it because keypress handlers were being used. This means that WebTest should probably use type(...) rather than setting the field’s value directly.

But this means as well that Google’s home page relies on the keypress handlers and that the JavaScript code doesn’t work correctly if these handlers are not called like in following scenario:

  1. – go to http://www.google.com/ncr (to be sure to have the English version of the page)
  2. – right mouse click in the search field to paste some value to search for
  3. – click the “I’m Feeling Lucky” button

The results are quite similar to the JS error obtained by WebTest, no matter if you use Firefox or Internet Explorer (perhaps does it work correctly with Chrome 😉 ):

Rather painful for Google than for WebTest!

OK, this error is nothing dramatical but for an Internet giant like Google that has plenty of talented engineers (and …, and…, and …. You perhaps know better than I what the web means for Google), I find quite painful to have such an error on its main page!

The error scenario is not particularly complicated and such a case should surely be covered by automated tests. Perhaps is it a question of inadequate tools? Google seems to use intensively Selenium 😉 .

Concerning WebTest, I’ll start the discussion on the mailing list to decide what is the best way to follow: setting directly the field value, typing it, or having both possibilities.

Advertisements

14 Comments

  1. Omkar said,

    September 12, 2008 at 4:17 pm

    I get the same error using HtmlUnit. How to avoid/getaround this error?

  2. lrbell said,

    September 15, 2008 at 3:19 pm

    So are there some working websites out in the real world that I can use for a DEMO? It worked fine in late July, then around early August it (google) started with JS errors…

  3. Marc Guillemot said,

    September 16, 2008 at 8:59 am

    Here is a code sample that allows to reproduce the problem and to get the JavaScript error:

    WebClient wc = new WebClient();
    HtmlPage page = wc.getPage("http://www.google.com/ncr");
    Thread.sleep(50);
    final HtmlTextInput q = page.getFirstByXPath("//input[@name = 'q']");
    q.setValueAttribute("HtmlUnit");
    final ClickableElement button = page.getFirstByXPath("//input[@value = \"I'm Feeling Lucky\"]");
    button.click();

    The Thread.sleep(50) is important here. Without it, the test will pass because some JS code started with setTimeout won’t have time to be executed.

    To avoid the problem, you need to use the type(…) method to trigger the handlers instead of using setValueAttribute(…):

    q.type("HtmlUnit");

  4. Omkar said,

    September 19, 2008 at 1:32 pm

    Mark,
    I understood how to AVOID the error(by using type(…) but cant I somehow suppress(not avoid) the error and let htmlunit proceed with processing the page?

  5. Omkar said,

    September 20, 2008 at 3:30 pm

    I am facing a problem when using htmlunit with DWR

    My htmlunit program is as follows:

    WebClient wc = new WebClient();
    HtmlPage page = wc.getPage(“http://xyz.com/xyz.html”);
    Thread.sleep(50);
    final ClickableElement button = page.getFirstByXPath(”//input[@name = \”testdwr\”]“);

    //this button click navigates to xyz.jsp containing the javascript
    button.click();

    The javascript is as follows:

    //file:xyz.jsp

    window.onload = send;

    function send(parameters) {
    dwr.engine.setActiveReverseAjax(true);
    SomeDWRClass.send(parameters), {
    });
    }

    //this function is called by DWR using reverse AJAX
    function serverresponse(data) {
    alert(data);

    //if i include the following call “processData” i get an error
    //if i exclude it no error occurs
    processData(data);

    }

    function processData(data) {
    //….
    }

    The error that i get in my htmlunit program is as follows:

    Exception in thread “HtmlUnit Managed Thread #1: XMLHttpRequest.send” org.mozilla.javascript.EcmaError: TypeError: Cannot find

    function append in object [object]. (script in http://xyz.com/xyz.jsp from (72, 9) to (539, 10)#257)
    at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3558)
    at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3536)
    at org.mozilla.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3564)
    at org.mozilla.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3583)
    at org.mozilla.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:3647)
    at org.mozilla.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2187)
    at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:3170)
    at script.chatReceived(script in http://xyz.com/xyz.jsp from (72, 9) to (539, 10):257)
    at script.serverresponse(script in http://xyz.com/xyz.jsp from (23, 9) to (58, 10):54)
    at script(http://ronin.rnd.avaya.com/website/servlet/chatconnection/engine.js#979(eval):2)
    at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:2484)
    at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:164)
    at org.mozilla.javascript.ScriptRuntime.evalSpecial(ScriptRuntime.java:2426)
    at org.mozilla.javascript.ScriptRuntime.callSpecial(ScriptRuntime.java:2278)
    at org.mozilla.javascript.Interpreter.interpretLoop(Interpreter.java:3223)
    at script(http://xyz.com/servlet/dwr/engine.js:979)
    at script(http://xyz.com/servlet/dwr/engine.js:868)
    at script(http://http://xyz.com/servlet/dwr/engine.js:645)
    at org.mozilla.javascript.Interpreter.interpret(Interpreter.java:2484)
    at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:164)
    at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:401)
    at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:169)
    at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3004)
    at org.mozilla.javascript.InterpretedFunction.call(InterpretedFunction.java:162)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:465)
    at com.gargoylesoftware.htmlunit.javascript.host.XMLHttpRequest.setState(XMLHttpRequest.java:142)
    at com.gargoylesoftware.htmlunit.javascript.host.XMLHttpRequest.doSend(XMLHttpRequest.java:413)
    at com.gargoylesoftware.htmlunit.javascript.host.XMLHttpRequest.access$000(XMLHttpRequest.java:54)
    at com.gargoylesoftware.htmlunit.javascript.host.XMLHttpRequest$1.run(XMLHttpRequest.java:360)
    at org.mozilla.javascript.Context.call(Context.java:502)
    at org.mozilla.javascript.ContextFactory.call(ContextFactory.java:511)
    at com.gargoylesoftware.htmlunit.javascript.host.XMLHttpRequest$2.run(XMLHttpRequest.java:366)
    at java.lang.Thread.run(Thread.java:619)
    at com.gargoylesoftware.htmlunit.ThreadManager$1.run(ThreadManager.java:127)

    Can you please tell me what is causing the error? Is my implementation ok?

  6. Marc Guillemot said,

    September 26, 2008 at 12:34 pm

    Omkar,
    I don’t understand what you make for a difference between *avoid* and *suppress*. You have to ask by Google, not by me, if you want that the page doesn’t produce any JS error.

  7. Marc Guillemot said,

    September 26, 2008 at 12:35 pm

    Omkar (2),
    this blog is not intended for issues with HtmlUnit. Please use HtmlUnit mailing list instead.

  8. Richard said,

    October 14, 2008 at 2:22 pm

    Works better for http://www.google.co.uk/

  9. Marc Adenaur said,

    January 7, 2009 at 11:12 am

    Amazing how limitations in your own software can be shown as painful mistakes of somebody other

  10. JW said,

    March 11, 2009 at 10:20 pm

    Marc,

    you don’t really know what you’re talking about, do you?

  11. Steve Pence said,

    June 30, 2009 at 7:31 am

    Hi,
    I just installed web test and I really, really need a sample that simply works, just to validate my install and see working tests and get going with this product. Mark, can’t you post a totally done sample against some other website, like maybe your own site? Just for install validation for us total newbies. It would be a huge help.
    Thanks much,
    Steve

    • EP said,

      August 13, 2009 at 4:14 pm

      +1 on what Steve P said

  12. james said,

    January 22, 2010 at 11:26 am

    Hi All,

    I am using Java WebClient to retrive my gmail automatically. Here following some sample lines:

    webClient = new WebClient(BrowserVersion.FIREFOX_3);
    webClient.setJavaScriptEnabled(true);
    loginUrl =”http://mail.google.com/mail/?ui=html&zy=c”;
    loginPage = webClient.getPage(new URL(loginUrl));
    System.out.println(“javascript enabled: “+webClient.isJavaScriptEnabled());
    if ((null == loginPage) || (HttpURLConnection.HTTP_OK != loginPage.getWebResponse().getStatusCode())) {
    log.error(“cannot access “+loginUrl);
    return;
    //System.out.println(loginPage.asXml());
    }
    loginForm = (HtmlForm) loginPage.getElementById(“gaia_loginform”);
    email = loginForm.getInputByName(“Email”);
    password = loginForm.getInputByName(“Passwd”);
    btnLogn = loginForm.getInputByName(“signIn”);
    email.setValueAttribute(testEmailAddress);
    password.setValueAttribute(testEmailPasswd);
    webClient.setJavaScriptTimeout(10000000);
    webClient.setRedirectEnabled(true);
    pageHome = btnLogn.click();
    if ((null == pageHome) || (HttpURLConnection.HTTP_OK != pageHome.getWebResponse().getStatusCode())) {
    log.error(“cannot login to gmail”);
    //System.out.println(pageHome.asXml());
    }
    System.out.println(pageHome.asText());

    However, I keep getting following error:
    JavaScript must be enabled in order for you to use Google Mail in standard view. However, it seems JavaScript is either disabled or not supported by your browser. To use standard view, enable JavaScript by changing your browser options, then try again. To use Google Mail’s basic HTML view, which does not require JavaScript, click here.If you want to view Google Mail on a mobile phone or similar device click here.
    Loading ihatecari@gmail.com…
    Loading standard view | Load basic HTML (for slow connections)
    1.
    2.
    3.

    Please help.

    Thank you very much in advance.
    James
    james@jdmail.me.uk

  13. October 11, 2011 at 9:07 am

    […] not good while running automated tests. HtmlUnit provides a great help for that (see for instance this) but often integration tests should run in “real” browsers and not in HtmlUnit (for […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: