On Friday of last week I found myself hacking on a peculiar bug that I stumbled upon in Popcorn.js. The particulars of that bug aren't very important, the only important thing to mention right now is that it was a bug only in Internet Explorer 9. For the better half of the day, I found myself running and re-running the tests in IE in a hopeless attempt to even get to the test I needed to run. The thing with IE is that it is incredibly unpredictable when running the Popcorn.js core tests. A test could pass on one run, but then never pass again. And don't even get me started on compatibility view (more like incompatibility mode) which basically breaks the internet by turning off the "addEventListener" method...
Eventually, I got fed up and decided then and there that I was going to do something about it. From my experience working on these tests over the last couple years, I know first hand that the biggest problem is how the test cases re-use the testing media throughout the majority of the suite. Another major issue with the tests is that a failure in one case can ripple through the rest of the suite, creating false negatives.
At that point, I could think of two solutions to the problem. The first would involve going through the 5000+ line test file, case by case, and figuring out how to ensure each could run without and worry about tests before and after it. What this would involve, is a foolproof way for each test to "clean up" regardless of a pass or fail. This would involve: resetting the media to 0 seconds, destroying any and all popcorn instances created during the test case, removing any DOM elements added during the test and removing any and all test plugins created for the purposes of the test case. That's a lot of work for over 130 sets of tests.
The second solution would involve "sand-boxing" each test case into it's own iframe, where it could run in an isolated environment and not have to worry about where it's starting or how it should leave the page after a pass or fail. This would be a lot of work as well, but in theory I could create a template page to use for each case, copy the case into a script tag on that page, and add/remove various dependencies (such as audio/video tags). As for running the tests, I already built the top-level test running framework which runs every test suite in the project. I'd need to make some changes to it, but the core functionality I needed was there.
All things considered, I chose to do the sand-boxing, simply because once it is completed, creating and debugging tests in the future should be a far more simple task than it is right now.
I started by creating a template page for each case to work with, and migrated the first 5 tests into their own files. From there, I turned to the test runner to see what modifications were needed. Turns out it was a little more complicated than I had originally anticipated. To get what I wanted done, without creating a separate runner, there were 3 things I needed to change:
- The runner needed to take a test configuration file as a parameter when it's being instantiated, allowing for the specification of different sets of tests.
- The way the runner parsed the file would need to be changed. The original test runner used query strings and check boxes to determine which tests to actually run, but in the core test case, every test should be run.
- Results from tests would need to be posted to parent windows. This would allow for the main Popcorn Test runner to execute the core test runner in an iframe and receive the results.
Naturally, I wanted to see how they performed in IE9. Turned out that many of the test that actually depended on previous tests letting the media (audio/video) load, would try to access things like duration, or try to change the currentTime of the media before it was loaded. The rest of Tuesday and a majority of Wednesday I worked to update these tests with appropriate listeners for readyState. I also changed a few of the tests from synchronous tests to asynchronous tests.
Besides having to fix a few HTML5/CSS3 errors, and some visual issues with the test runner, By mid Wednesday, I had completed my work. For the first time in Popcorn.js testing history, you could reliably run our core unit tests without getting errors. I say that after testing on:
- Firefox 18 (Mac & Windows 7)
- Chrome 24 ( Mac ) There's a problem on Chrome 24 for Windows 7, The browser crashes because of a bug where it's creating too many media decoding threads and not cleaning them up fast enough. I can get them to pass if I pause between each test for about 1 second.
- Safari ( V6 on Mac, 5.1.7 on Windows 7)
- Opera 12.12 (Mac and Windows 7 )
- IE 9 (Windows 7)'
Links: