For the past 3 weeks I have been working on designing an automated user interface testing system for Butter.js. In this post, adapted from a recent email I sent, I will describe all the components of the system and how one can go forwards with creating tests. The system I designed uses a testing framework called Selenium, and a distributed testing system known as Selenium Grid. For automating the tests, I will leverage GitHub's post-receive URL functionality, allowing tests to be executed on a per commit basis. There's still plenty of work to do, but I'm sure once it's done we'll have a solid system for ensuring that we never break something without knowing!
Selenium Grid is a Java based testing infrastructure that relies on a central server, called "The Hub". Connected to this hub are separate entities, called "nodes". When Selenium receives a connection requesting tests to be run, the hub will look at the request, and based on what browser/os and what nodes are currently connected, the hub will forward the test commands to the node. Upon receiving tests from the hub, the node will launch the requested browser, perform the tests and return the results back to the hub. The hub will then send the results back to the script that initiated the tests. Nodes are handy because (once I can figure out how to configure everything properly) we'll be able to run tests on different platforms and different browsers. You can read more about Selenium Grid here.
Selenium tests can be written in several different languages ( C#, Ruby, Java, etc. ) but I have chosen to use Python. For our purposes, tests will be grouped into python files by type, i.e. TrackEvents, Menus, Save/Export. Inside each python file, we will place the test cases. Each case is defined as a class, for example "testcase_TrackDrop" would be a good name for testing dropping a track onto the timeline. The python module that Selenium provides gives us a multitude of useful commands for creating tests (more on that later). Simple things like checking for an element's existence are possible, as well as specifying clicks, drags and drops, to even more complex things like injecting JavaScript that can perform some function and return a value to selenium. The basic structure of a test case would be to perform a list of actions, and enforce several assertions before, during and after executing actions on the UI. If all of the assertions return true, selenium will record the test case as a Pass. You can check out the python API here.
Writing tests shouldn't be too difficult, even for those who are less familiar with python ( First time I ever used it was about 2 weeks ago ). Selenium provides an extremely useful Firefox add-on called Selenium IDE that enables tests to be prototyped very easily ( although there is a slight learning curve ). With the IDE you can specify the actions you wish to perform, selecting what to perform them on in the webpage and where you want them to be performed. It also allows you to run the tests live on the target web page, so it's super easy to debug the tests as you develop them. Once you've written your tests, the IDE gives you an option to export them into Python. You can read about & get the Selenium IDE here.
After exporting the test into Python integrating it into the test system I've been developing will be straightforward and simple. You must determine if the test belongs to an existing set of test cases (like I described earlier) or if you need to create a new file for the type of test you have created. I'll go over both types for you guys.
If you are adding a new Python file for the test case, add this header to the top of your Python script:
This code sample was lost because wordpress
Ensure the "setUp" method of your test case class looks like:
This code sample was lost because wordpress
...which grabs the necessary configuration variables for Selenium, and opens the connection to the grid
Copy the meat of your test (read: test command & assertions ) into a "test_case" method
This code sample was lost because wordpress
Make certain to tell selenium to open the right test page using "testvars['ButterTestPage']"
You should then fork and clone my ButterSeleniumTesting repository on GitHub.
Within the 'contents/' folder is a file called "testRunner.py" which needs to be made aware of the new test case(s).
If you created a new file, import it below this bit of code:
This code sample was lost because wordpress
If no new file was created skip the above step
Add the test to the test runner. Use the current lines there as an example of how to load them.
This code sample was lost because wordpress
You should now be good to go!
The team will have to decide on a proper way to review and approve tests as we move forward. Perhaps this could be done by creating a "try" server of sorts.
Another large component of the system is reporting test results. I was directed by Selenium's very sparse documentation to use a completely undocumented Python tool called HTMLTestRunner for doing this. After many hours of cursing and pacing back in forth in thought, I managed to get my code to generate easy to read static html report pages. Each test job that the grid gets will output a timestamped html file, and overwrite an existing "last_run.html" file that can be viewed to see what passed and/or what did not pass. These static report pages are helpful but are ultimately not quite what we want to be using in the long term. Here's a sample report page from an actual test run. I have a couple ideas for upgrading this process. I could modify HTMLTestRunner to post data to a PHP script which would record the results in a database, and have another page that can query the database for test runs and results ( this is probably the more complex solution ). The second idea is a little more simple. I'll changed the titles of the test result pages to commit ID's and create page that would scrape the results directory and list each test run in chronological order.
Another consideration will have to be maintenance. The test system I created needs to clone a new butter repository for every job. This will inevitably use up a lot of disk space, so It will need to have a mechanism in place to keep the disk usage in check.
There you go! A not so brief description of what's been done! Feel free to reach out to me if you have any questions, concerns or ideas. Thanks for reading!