Time to Automate Butter Unit Testing

For the past 5 weeks, the Popcorn team has been churning out hundreds upon hundreds of tickets in preparation for last weekends' Popcorn Story Camp. While ultimately satisfying that we achieved our goal, every team member was absolutely drained of all energy by the end of last week. It's something we're looking to never repeat. In order to prevent another month of endless ticket queues, regressions, hot fixes and late nights at the office, we're going to try to examine our processes and re-write our roadmap to 1.0. By putting together a new roadmap, we'll be able to look back at what we've achieved since January and set clear and attainable goals for each of our remaining milestones.

This project is an amazing one, and our app is ground breaking, and I'm sure every one of us is immensely proud of what we've created. But there's one thing we are severely lacking which would help us to stand 100% behind our code. That missing piece is unit testing. We do have some unit tests, but they cover a very small slice of code. The core of the problem is that we're not emphasizing the need for tests enough. We've been landing feature after feature without writing a single unit test to ensure they work, and do not break in the future. Due to that failure in demanding tests be provided, we get burned over and over again when a feature is modified or expanded, and breaks previous functionality.

I've been tasked to assist in changing this "tests are optional" culture we've grown to hate so much. I won't be focusing on writing tests, and I won't be checking each commit for tests. What I am going to do will not only enforce that every single commit has a full suite of tests run on multiple browsers and multiple operating systems, It will also encourage our developers, myself included, to run the tests before we try to merge the code into our master branch.

This system will rely on a couple of different components. The first component will be Bot.io . Bot.io is a "Fully scriptable build/test bot for Github projects". We currently use this tool in our Butter pull requests for previewing the app live and for running JSHint and CSS Lint on our code. I'm going to be creating new command scripts that will interface with the second component of the system in order to queue unit test runs on specific branches of our code.

The second component will be Test Swarm. Test Swarm provides "Distributed continuous integration for Javascript". For example, I will enable a whitelisted developer to comment on a pull request: "/botio try" and botio will command the swarm to queue unit tests on all of it's connected clients. Clients are simply browsers that I will set up in Virtual machines that will run any unit tests that the swarm has available. Eventually, I wish to expand on the syntax, so something like "/botio try -tests {type}" will tell botio to only ask the swarm to run the core unit tests. This could even be expanded to target specific browsers and platforms.

To make the above possible, I'll need to efficiently manage my time and set goals for myself. Here it goes:

• Bot.io
  • Write command scripts
  • When code is merged into master, it should trigger botio to tell the swarm to run all tests
    • explore if I can post in the related pull request, a link to the new job
  • Explore possibility for having botio post if tests fail
  • Add Try command for on demand test runs.
• Test Swarm
  • Creation of user accounts and public joining of the swarm needs to be disabled
  • Only Botio will have the credentials to run Test Swarm jobs
• Servers and  Test Environments
  • Most of the work is done
  • All three servers will need to have VMs with different browsers running.
  • Performance of these servers and their respective VM's will need to be monitored.
  • Small chance we might need more test machines for better coverage and to distribute the work load.
• Cleanup Jobs
  • Disk space is limited, and Butter and Popcorn Maker will quickly eat it up, so cron jobs will need to be set up to run several times daily.
    • They will delete any folders and files that are over 24 hours old on the botio machine and on the Test Swarm machine.
    • Might make sense to have one cron job run on botio, and command it to tell the swarm to clean up (synchronize jobs)

Thanks for reading!