Thoughts on Selenium Grid

I’m impressed by the activity around Selenium. Not because I’m impressed by Selenium but because I think that all this work could be better invested. For instance it would surely be a smaller effort to make for instance HtmlUnit able to work with its recognized speed with numerous browser configurations, simulating perfectly their JavaScript execution, and not only 2 browsers as Selenium really supports. The last Selenium related project I’ve discovered is Selenium Grid.

Run tests in parallel

Selenium Grid’s aims to allow running tests on multiple computer because:

  • Selenium remote control is quite slow at driving the browser“.
  • You can only run a limited number of concurrent tests on the same remote control before seriously impacting its stability. Practically speaking, launching more than 6 browsers on the same Selenium Remote Control is not advisable. The limitations are even more drastic for Internet Explorer.

In other words: slow and not scalable. If I correctly understand the documentation, Selenium Grid will allow to run approximatively 6 x N tests in parallel, where N is the number of computers in the grid. Not really impressive.

An HtmlUnit user reported that he performed load testing with HtmUnit using 350 threads in the same VM and one WebClient instance per thread. For this purpose he allocated 512M of his 1Gig laptop to the VM and it worked very well. WebTest surely adds some overhead to HtmlUnit “pure” but, due to the indications of this user, I could imagine that it would be possible to run 500 threads of WebTest tests on one computer within a single VM. This would save just… 80 computers compared to Selenium Grid!

Parallelisation often limited

Selenium Grid’s documentation says: “Selenium Grid cuts down on the time required to run a Selenium test suite to a fraction of the time that a single instance of Selenium instance would take to run“. Sadly this isn’t that easy. Not all tests can be run in parallel: as soon as you test the functionalities of registered users, you’re often limited to a serial test execution (or a just a few parallel executions) when your application doesn’t accept numerous simultaneous login of the same user and you don’t have so much users available with the right settings for the tests. In this case only pure speed can help.

Grid computing is an exciting large area, perhaps is Selenium Grid only a side product of some more significant activity…

Advertisements

10 Comments

  1. October 27, 2007 at 3:49 am

    Hi Marc,

    Thanks for sharing your thoughts. I am the principal author for Selenium Grid šŸ˜‰

    I totally agree with you that in process web testing will always be faster and simpler that running a web acceptance test in a browser. Just like running a unit test (done right) is always going to be faster that running an integration test involving a database and a messaging system. Moreover a lot of the view logic can be tested this way (target url, status, html rendered, etc.)

    As a matter of of fact I use in process web testing extensively myself in the form of Rails controller and integration tests
    (see http://weblog.jamisbuck.org/2006/3/9/integration-testing-in-rails-1-1 and http://wiki.rubyonrails.com/rails/pages/HowtoFunctionalTest if you need more details). I love it, and abuse it!

    Nevertheless there is also a place for Web Acceptance Testing involving a browser. Just like you are better off running some acceptance tests even if you have a very good unit test coverage. Moreover some web application features are actually quite hard to test without a browser (e.g. AJAX and browser specific workarounds). So a disciplined programmer will use both in process and browser based web testing. I believe these two techniques complement each other, the hard part is finding the right balance and using them wisely šŸ˜‰

    I would also like to clarify that Selenium Grid does not scale in a 6 x N basis as you describe. Sorry if the documentation is confusing. Selenium Grid actually scales with the number of Selenium Remote Controls (processes), not the number of machines. So the formula is more like 1 x P, where P is the number of remote controls running (potentially on multiple machines). Typically you can easily run 10-20 remote controls on a Mac Mini or an old PC. Of course your point is still valid, running all these tests in the same process using multiple threads is simpler and faster… by orders of magnitude! Nevertheless I do not want people to believe that their powerful 4-CPU server will only be able to run one Selenium test at a time šŸ˜‰

    I believe the tools we use for Web testing can still be dramatically improved. Ultimately HtmlUnit, Ruby on Rails outstanding controller testing and Selenium are all pieces of the same puzzle. Let’s keep pushing them further!

    Cheers,
    Philippe Hanrigou

  2. Marc Guillemot said,

    October 30, 2007 at 12:49 pm

    Salut Philippe,

    Does it mean that Selenium Grid allows to run 10-20 tests in parallel on a Mac Mini or an old PC (as long as so much profile are configured)?
    How does it look like on a powerful 4-CPU server?

  3. gnuyoga said,

    June 25, 2009 at 4:13 pm

    @Philippe Hanrigou

    thanks for your enlightining words on formulae for selenium grid.

  4. September 19, 2009 at 7:52 am

    Hi Marc,

    I just stepped into this and while I’m quite frustrated nowadays by the slowness of selenium in executing tests I’m also a bit curious to know how can you be so sure that HtmlUnit simulates perfectly the behaviour of real browsers (especially in terms of JavaScript / AJAX)?

    Br,
    Roberto Fasciolo

    • Marc Guillemot said,

      October 11, 2009 at 7:59 am

      Hi Roberto,

      in part of HtmlUnit build we have a really large number of unit tests to ensure that it behaves like real browsers. Additionally we run test suites of major AJAX libraries as part of our build too and expect to behave like the real browsers we simulate, meaning that a test should fail with HtmlUnit when it fails with the simulated browser. All in all, it’s already quite good… but surely not enough. I think that we’ll need in the future to write some tools to prove that HtmlUnit behaves like the browsers it simulates.

      What would be for you the proof that HtmlUnit simulation is good enough?

      Marc.

  5. Damian Martin said,

    January 19, 2010 at 6:46 pm

    We are looking into load testing our product but against what i have read it seems difficult to get the thing working. We are writting the tests in C# and I can not work out how to get tests running in parellel. 1 at a time is fine. There does not seem to be any documentation on get it to work. We are looking at having to start with 1,000 users, increasing to 10,000, 100,000 and eventually 250,000. Im not sure if this product is able to do this. Any help would be great

    Thanks
    Damian

  6. Anish said,

    May 13, 2011 at 6:54 am

    Hi,
    I am working on selenium RC based testing ,but we have hit an issue with the multiple browser testing.We are using Testng to drive our testing and we use java for developing our test scripts .I believe that by using Grid we can overcome the issue with mutiple brwser settings.

    Could you please give me some pointers on how to go about this in terms of setting up GRID and is there any possible impact on the exisiting tests written for RC

  7. Anish said,

    May 13, 2011 at 6:55 am

    Thanks in advance

  8. Bassim said,

    January 24, 2012 at 8:14 am

    Hi Anish,

    Since you are already familiar with TestNG then setting up the selenium grid is a piece of cake, just follow the instructions on the selenium Grid’s documentation page. However, I should point out that in the testng.xml file in the suite tag you should specify that the test should run in parallel like this,

    …. here you tests tag come.

    There is also a really good tutorial that I stumbled upon you should take a look at it.

    http://technologyandleadership.com/six-steps-for-complete-test-automation-with-selenium-grid/

  9. March 26, 2017 at 11:35 am

    Hi Anish,

    Thanks for a nice article on TestNG. I know the configurations and setting Selenium Grid is grey hairs for me. Your article been a great help while working with grids.

    I have tried my best to pen down a guide on Selenium Grid and TestNG installation and configurations. Being an expert in Selenium, i would like to request you to please spend a moment to provide your feedback.

    Selenium Installation Guide

    Thanks in advance my friend.

    Regards,
    Sherlin


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: