Late last week I began working on something that we'd needed on Popcorn since even before I started at CDOT last May. For the longest time, when testing for a release, we would have to manually click through all of the plug-in, player, parser and module unit tests. This is a very tedious process, as one might expect. Now, we did have a rudimentary test runner for the plug-in directory, but it didn't record and tally any test results. This meant that if you looked away for too long, you could miss a test's result entirely.
I began my task with several key goals for the task:
- The test runner should look & feel like QUnit (QUnit is the JS testing framework we use).
- The test runner should provide results about each test ( Passed, Failed and Total Run ).
- There should be a link to each individual test, so that the user can navigate to it in the event of a failure.
- There should be a way to view more information about each test (I.E. expand each test result to view more detail.)
Look and Feel
Making the test runner look and feel like QUnit was simple to accomplish. I copied the core unit test file into a new one, and stripped out the cruft. figuring life would be easier, I just kept the default QUnit CSS file. I then added an iFrame to the page for loading the test pages. From that point on, I needed to make sure that I built the results properly in my JavaScript code, apply the correct classes, and append them to the page. If all went well it'd work perfectly.Building the Test Runner
QUnit provides a mechanism to define methods, which, at certain points in the testing process, are called. An example of this is `testDone`. When a block of assertions completes, QUnit will call the `testDone` method. I took advantage of this by creating a script called `popcorn.inject.js`, which defines two methods: `testDone` and `done`. The latter of the two is called on the completion of all assertions and tests on a QUnit page. In each of these methods, you receive data about the tests that are passing and failing. Using `postMessage`, I can push the results to the iFrame's `parent window`. I made sure that this won't execute if there's no parent window, in the event the test is run individually.The structure of the test runner is fairly straight forward and simple. I made an array of JavaScript objects that describe each test. Here's what it looks like:
This code sample was lost because wordpress
On page load the runner attaches a postMessage handler to the iFrame. The runner will then iterate over the array of objects one at a time. It loads the first test into the iFrame, kick-starting the entire process. As the test in the iFrame executes, it posts messages back to the parent window. My postMessage handler will detect whether the message is a testDone or a done message type. If the message is testDone, it pushes the results from it onto an array. When it receives the done message, It will build the necessary elements based on the array of data collected from testDone events. It then appends them to an outer element, that contains totals about how many tests passed and failed, along with the time it took to ran the tests. These totals are added to global totals I created ( Not global as in attached to window, but global to the test runner :P ), which track passes, fails, totals and run time across all tests that the runner executes. Once it's finished building the result element, it appends it to the test page, and loads the next test in the array. When it has exhausted all of the defined tests, it will output the global results of all run tests, and the total time it took to complete everything.
Once all of this was completed and working, I decided to go one step further in my work. Not only did I create a plugin test runner, but I was able to create a test runner for every area of our code. That includes runners for Parsers, Players, and Modules. And for the first time ever, a test runner for all 1600+ tests in the project!
Below I've attached screenshots of the before and after transformation of the test runner (click to enlarge)
[caption id="attachment_241" align="alignnone" width="400"]
This is what the test runner used to look like[/caption]
[caption id="attachment_240" align="alignnone" width="400"]
This is the new Test Runner :D[/caption]
What Does This Mean?
The implications of the work I've done here are pretty significant. By implementing these test runners, the time spent manually testing for releases should drop dramatically. For example, I remember many releases where I had to navigate between the eight parser unit tests manually for at least 4 browsers, on multiple platforms. It was tedious work, for sure. But with the new parser test runner, you can automatically run all the parser tests in 4-8 seconds, and have the results reported back to you cleanly and efficiently.I really hope the community loves the new test runners, I know I already do! Moving forward, I will focus my efforts on developing the integrated testing system for Popcorn.
Cheers!