Let's say you have one hundred files in the "src" directory, and you need to upload them to S3. You could add a resource entry for each file, but that would be tedious and repetitive:

locals {
  src_dir      = "${path.module}/src"
}


resource "aws_s3_bucket_object" "index" {
  bucket        = local.my_bucket_id
  key           = "index.html"
  source        = "${local.src_dir}/index.html"
  content_type  = "text/html"l
}

resource "aws_s3_bucket_object" "about" {
  bucket        = local.my_bucket_id
  key           = "about.html"
  source        = "${local.src_dir}/about.html"
  content_type  = "text/html"
}

resource "aws_s3_bucket_object" "main_css" {
  bucket        = local.my_bucket_id
  key           = "main.css"
  source        = "${local.src_dir}/main.css"
  content_type  = "text/css"
}

resource "aws_s3_bucket_object" "javascript" {
  bucket        = local.my_bucket_id
  key           = "main.js"
  source        = "${local.src_dir}/main.js"
  content_type  = "application/javascript"
}

resource "aws_s3_bucket_object" "favicon" {
  bucket        = local.my_bucket_id
  key           = "favicon.ico"
  source        = "${local.src_dir}/favicon.ico"
  content_type  = "image/x-icon"
}

resource "aws_s3_bucket_object" "header_image" {
  bucket        = local.my_bucket_id
  key           = "header.png"
  source        = "${local.src_dir}/header.png"
  content_type  = "image/png"
}

# and so on, 90+ more times

Thankfully, Terraform has the for_each meta-argument which lets us pass a map or set of strings to create an instance of a resource for each element in the meta-argument. Pairing this functionality with the fileset built-in function allows us to generate all these objects in far fewer lines of configuration:

locals {
  src_dir      = "${path.module}/src"
}

resource "aws_s3_bucket_object" "site_files" {
  # Enumerate all the files in ./src
  for_each = fileset(local.src_dir, "**")

  # Create an object from each
  bucket        = aws_s3_bucket.bucket.id
  key           = each.value
  source        = "${local.src_dir}/${each.value}"
  
  # Uh oh, what should we do here?
  # content_type  = ???
}

There is one small problem though, as indicated above: content_type. How can we set the content type correctly for each file? Well, there's a few built-ins to help us out here. Firstly, there's the lookup built-in, which returns the value from a map given a key, and can set a default if no key is found. So, if we define a content type map like so:

locals {
  content_type_map = {
    html        = "text/html",
    js          = "application/javascript",
    css         = "text/css",
    svg         = "image/svg+xml",
    jpg         = "image/jpeg",
    ico         = "image/x-icon",
    png         = "image/png",
    gif         = "image/gif",
    pdf         = "application/pdf"
  }
}

We can use lookup get the content type by extracting the file extension from the filename. To accomplish that, we use the regex built-in function:

regex("\\.(?P<extension>[A-Za-z0-9]+)$", filename).extension

So, if we put it all together, we get:

locals {
  src_dir      = "${path.module}/src",
  content_type_map = {
    html        = "text/html",
    js          = "application/javascript",
    css         = "text/css",
    svg         = "image/svg+xml",
    jpg         = "image/jpeg",
    ico         = "image/x-icon",
    png         = "image/png",
    gif         = "image/gif",
    pdf         = "application/pdf"
  }
}

resource "aws_s3_bucket_object" "site_files" {
  # Enumerate all the files in ./src
  for_each = fileset(local.src_dir, "**")

  # Create an object from each
  bucket        = aws_s3_bucket.bucket.id
  key           = each.value
  source        = "${local.src_dir}/${each.value}"
  
  content_type  = lookup(local.content_type_map, regex("\\.(?P<extension>[A-Za-z0-9]+)$", each.value).extension, "application/octet-stream")
}

And there you have it! all the files in your "src" directory should now have an associated S3 resource managed using terraform, and each has the appropriate content type, so you can serve the files up via S3's static site functionality or via a CloudFront CDN.

I've just made my latest project public: Autoku

(Watch closely above, you might notice a little guest!)

Autoku is a command line tool for creating and configuring Heroku applications using configuration files i.e. "Infrastructure as Code". It allows you to specify exactly how your application should be configured, and modifies it to reflect that.

The tool has reached a point where I'm comfortable sharing it, but be forewarned that it's still in need of plenty of work. There are some quirks that need to be worked out and plenty of automated testing to get done. Seeing as this tool is open source, I welcome any and all contributions.

Install

yarn global add autoku or npm install -g autoku

Usage

Given this yaml file sample.yaml:

name: sample-app

region: us

maintenance: false

stack: cedar-14

configVars:
  SOME_VAR: foo 

addons:
  heroku-postgresql: hobby-dev

collaborators:
  - person@example.com

features:
  - log-runtime-metrics
  - http-session-affinity

formation:
  web:
    quantity: 1
    size: hobby
  worker:
    quantity: 1
    size: hobby

logDrains:
  - https://example.com:7000

domains:
  - mydomain.example.com

sni:
  - certificate-chain: "-----BEGIN CERTIFICATE----- ..."
    private-key: "-----BEGIN RSA PRIVATE KEY----- ..."

buildpacks:
  - heroku/python

Execute autoku deploy ./sample.yaml -k $HEROKU_KEY to have Autoku create (if needed) and configure your application.

Subsequent calls will update the application if sample.yaml changes.

Using Autoku you can maintain the following parts of your Heroku app using configuration files:

• maintenance status
• Configuration Variables
• addons
• collaborators
• platform features
• formations
• log drains
• domains
• sni endpoints
• buildpacks

Again, this is a very new piece of software and so it has flaws. There's plenty of problems to solve, like how to store and retrieve sensitive variables, for example. On that note: don't use an Autoku config file to store secrets in a public repository

Configuring and running services for Webmaker is a real pain in the ass.

(Webmaker services = Webmaker API, Webmaker ID, and Webmaker LoginAPI)

There's database and caching services to install and configure, npm dependencies to install, native NodeJS bindings to compile (sorry, Windows), and database scripts/migrations to run. This makes life difficult for many. Particularly those who don't handle these things on a day to day basis, like front-end focused developers or designers. I wanted to make life easier for them, so I took some time to fix that problem.

I decided to fix this problem using Docker. Docker is a platform for building self contained applications that have their own filesystem, runtimes, system libraries, and more. What this basically means is that you can build a Docker image once, and expect it to run anywhere (...that can run Docker).

Webmaker-Docker is a new repo I've created to contain everything you need to run Webmaker Services, without needing to install anything except git, Docker, and Docker Compose. That's right, no need to install NodeJS, npm, Postgres, MySQL, SQLite, Redis. No need to run npm install or bower install. We're finally living the dream, folks.

This development will enable something we've never had before: The ability to give front end devs and designers simple and easy access to services they can use when prototyping and developing new features. It's even possible to create dockerized feature branches for team mates to use new service features to develop front end parts in paralell! (Follow up post on that another time)

How To Run

Okay, so you're super excited about this, but not sure exactly what to do next. Let's go over the set-up (subject to changes :P). I should note that docker will automatically download container images from Docker Hub, so make sure you've got a decent internet connection or time (or both) - but don't worry, this will only happen the first time.

1. Install Docker and Docker Compose (available for Linux, Mac and Windows) - Installation instructions
2. Clone the git repository: git clone git@github.com:cadecairos/webmaker-docker
3. Go to the data-services directory and run docker-compose up -d to start the database services Webmaker needs.

• They're started in the background by passing the -d flag, omit it if you want to see container logs.

4. Go to the services directory and run docker-compose up -d to start the Webmaker services.

• omit -d to see logs from the Applications (helpful for debugging)

5. To shut down the containers when they're detached, go to the two directories mentioned above and run docker-compose stop, otherwise, ctrl-c does the trick.

Now what?

That's up to you! You've got fully functional services running that you can use to do whatever you please. Wanna prototype a new feature or site that depends on one of these services? Go for it! Here's a rundown on what services are exposed, and where:

• Postgres is listening on localhost:5432
  • username: 'webmaker'
  • password: 'webmaker'
  • database: 'webmaker'
• MariaDB is listening on localhost:3306
  • username: 'wmlogin'
  • password: 'wmlogin'
  • database: 'wmlogin'
  • root password :'root_wmlogin'
• Redis is listening on localhost:6379
  • There aren't any credentials to worry about
• Webmaker API is listening on localhost:2015
• Webmaker ID is listening on localhost:1234
  • there's a client and secret already in the database for you
    • client_id: 'webmaker'
    • secret: 'webmaker'
    • all grant types and response types allowed.
    • redirect_url is 'example.com' - You can manually change it or insert a new one if you desire
• Legacy Login is listening on localhost:3000, but don't touch it if you know what's good for you.

Let's get technical

Let's talk about how this all is put together. The base of all this magic is two files known as Compose files. Compose files use YAML to define a collection of Docker Images to build/download/execute. It defines what ports to expose, what kind of networking to use and lets you pull in env files to configure the applications running within.

Here's the data-services Compose file:

Here's the Webmaker services Compose file:

Each of the services is put together with a Dockerfile. Dockerfiles are text files that automate the creation of images. Here's the Dockerfile for Webmaker API:

There's plenty more to find in the repo, so go take a look!

Recently, I've been working with a new framework called Hapi to build an API for Webmaker. This is a bit of a departure from the past, where we traditionally would have used Express to build the our server applications. The decision to use Hapi was based on several features that we found in our experimentation with the framework. I'd like to outline these features, and give examples about how we're using them.

Tests

We wanted our services to be highly testable. Hapi's Server API makes this a cinch. Its configuration-centric approach to building servers means you can split all of your configuration (like routes) into require-able modules that can be tested in isolation:

As you can see above, we can test the routes' configuration outside of actually building a running Hapi server. While the tests above don't cover situations where more configuration is added, you can use libraries like Joi to provide far more strict assertions on the configuration object.

One other key Hapi feature is it's inject function, which lets you simulate receiving a request. It is invaluable when testing, because it enables you to do very cool things like providing credentials that step over the authentication of your routes.

Plugins

Hapi provides a plugin API, which makes separating your application into independent units very easy. This separation consequently makes testing really easy too. In your tests, you can register the plugin on a bare Hapi server, with whatever test specific configuration you desire, and test it's behaviour in isolation.

Server Methods

In the last gist I embedded, I added something called a server method. Server methods are a way to expose functions on your server object, which removes the need to require a common module everywhere a function is needed. Basically, if you define your server methods in a plugin, you register it once, and it's available everywhere!

Another really handy feature that server methods have is caching. Hapi is compatible with catbox, a multi-strategy key-value store, and Hapi leverages it for easy caching. This is extremely useful if the server method requests data from a database:

Validation

Hapi provides an interface for enforcing strict rules on the data coming into your application. This validation functionality works perfectly with the Joi library. It can be applied to route params (/foo/{bar}) request payload ({foo: 'bar'}) or query params (/foo/bar?fizz=buzz).

In summary, I'm very impressed with Hapi. With it (and with the help of a couple other great libraries called sinon and nock), I was able to achieve 100% test coverage on api.webmaker.org. All without having any external dependencies (other than PostgreSQL, but I can live with that, since the tests feel more real if they use an actual database)

Here's the part where I engage with you:

Do you use Hapi?

What do you think of it?

What are your favourite features or tricks when developing applications with Hapi?

edit: I didn't realize until this morning that a recent theme update disabled disqus. It's working now, should you wish to chat.

In my previous post, I wrote about the new login system we're working on for Webmaker. In short, the new system facilitates the authentication of a user by generating a one time use password and sending it to the user's email account. The user can then click a link in the email to log them in right away (either for the session only or for one year).

After getting some great feedback about my last post, I'm going to try and outline the protocol with less noise and more of the important details.

What's new

The latest implementation of the system adds in several things that weren't in the previous iteration.

Most notably, we've chosen to provide users with the option to disable one time passwords and use a custom set password. You can read more about how these are implemented in the Protocol section below.

Secondly, I've updated the OTP generation process to create ten character, pronounceable tokens. This new method aids users if they need to type the token on one device, while reading from another.

Lastly, I have put in rate limiting middleware on the login server routes that handle the new protocol. It is backed by redis, and provides a simple way to control how often someone can use the routes.

Here's how it works:

1. A request comes in!
2. Data about rate limits is stored in redis, keyed on "{API_route}:{IP_Address}:{uid}". The middleware queries redis to see if this IP and uid have been here recently, and if so, increments the hit count OR The rate limiting middleware stores the hit count(1) in redis, using the key pattern described, and the key is set to expire after some amount of time.
3. Should further requests be made from the same source, the count stored in the key is incremented
4. If the count in the key reaches some max value before it expires, any further requests get a 429 response from the API server (until the key expires).

High Level Overview

I've put together four high level flow charts that show the new actions in the system. For more detail about the actions, read the Protocol section below.

1. Create Account Flow
2. Sign In Flow
3. Enable and Disable Password Flow
4. Password Reset Flow

Protocol

Creating a token

1. A request is made to /api/v2/user/request with a JSON post body containing a uid (username or email)
2. The server generates 4 random bytes using node's crypto.randomBytes function
3. A module called proquint turns the random bytes into a pronouncable string, i.e: "joban-ladim"
4. The string is stored in the database, and an login email is dispatched

Sign in with Token

1. The user enters clicks a link in a login email. The link includes the following query parameters:
   • username=<username>
   • token=<OTP>
   • validFor=<'session' or 'one-year'>
2. The page issues a post request to /api/v2/user/authenticateToken with the query params as JSON in the body
3. The token is verified, and marked as used.
4. The server sends a set-cookie header in the response to the client, that either expires when the browser session ends (public/shared computers), or one that will not expire for one year (for trusted computers)

Sign in with Password

1. The user enters in their uid (email or username) and their password, which are posted to /api/v2/user/verify-password
2. The server looks up the user's salted and hashed password, and verifies that the provided password produces the same output from bcrypt
3. The server sends a set-cookie header containing the user's session object, which can expire after the session or in one year.

Enable Passwords

1. A logged in user provides a unique, secure password (ideally) which is send in to api/v2/user/enable-passwords in the post body.
2. The server uses bcrypt to salt and hash the password (12 rounds, to slow things down a bit)
3. The salt and hash are stored in the database

Disable Passwords

1. A logged in user causes the site to post to /api/v2/user/remove-password
2. The user's password is removed from the database, enabling OTP for the user once again.

Reset Password Request

1. A user provides a uid (email or username) which is posted to /api/v2/user/request-reset-code
2. The server uses Hat to generate 256 random bits, which is base 16 encoded, creating a 64 character long string.
3. The string is saved to the database and an email is dispatched to the account owner.

Reset Password

1. A user clicks the reset password link in a reset request email, which links to something like /reset-account?uid={username}&code={reset_code}
2. The user enters a unique, secure password (every time, right?)
3. The code, uid, and password are posted to /api/v2/user/reset-password
4. The reset code is validated
5. The same steps described above are followed to salt and hash the password, which is then stored in the database.

Wrap-up

The system we've been working on is an effort to make the sign up and sign on experience on Webmaker easier for our users. We think we've nailed it with the dead-easy sign up flow. However, getting sign in to not suck isn't as easy. As we roll out the new system in the near future, we're going to be watching closely and gathering data about the various ways people interact with the new system. This data will help inform our decisions moving forward as we try to make the Webmaker experience better for everyone.

If you've made it this far, congratulations! Do you have an opinon on the new system? We'd love to hear from you! There are many ways to get in touch:

1. Use the Disqus comments at the end of this post
2. Tweet @ me
3. Come talk to us in IRC - irc://irc.mozilla.org/#webmaker (I'm "cade")

Webmaker users currently sign in to their accounts using Persona, Mozilla's privacy respecting authentication system. It's fairly simple, and has worked really well since our rewrite this past march. You can read the details of the implementation in the blog post I've just linked.

Today, we're experimenting with alternative methods for users to log in and sign up. One such method, which Matthew Wilse has built prototypes for, has been called the "Passwordless" or "Handshake" method. I would argue that calling the system passwordless is a bit misleading and that handshake is not clear about what it really is, so I'll refer to it from here on out as the One Time Password(OTP) system.

For the past several weeks I've been building this system into existing Webmaker applications. We're hoping to deploy it to a limited number of people and gather some feedback and statistics, particularly whether or not it improves successful signup/signins (Persona is great, but a lot of users just give up trying to make it work).

Another benefit of the work I've been doing is a huge improvement on the account creation process. Here's some background on how it currently works:

1: A user clicks "Sign Up" somewhere on Webmaker
2: A persona pop-up appears
3: A user who does not have a Persona account must now create one. If they have one already, Go To 5
4: Verify email with Persona.org
5: Sign into Persona
6: A new user modal pops up, Asks for a username, among other things.
7: Yay, a new user!

At every step above number 7, some significant percentage of users just simply gives up. I don't blame them, it's incredibly confusing to have to create two accounts as well as verify an email address.

Matthew's prototypes provided a sign-up flow that looked a little bit like this:

1. A user clicks "Sign Up" somewhere on Webmaker
2. They provide an email and a username
3. Yay, a new user!

Wow, that sounds really nice! So I'](GHOST_URL/content/images/sign-up.gif)

I've also made the OTP sig](GHOST_URL/content/images/sign-in.gif)

During the early stages of this work, I build the front end bits as best I could, but my CSS-Fu has never been that strong. Recently, Ricardo Vazquez has come on board the login train to make the prototype modal dialogs beautiful to see and use. Stay tuned to my blog and Webmaker demos in the near future to see the project evolve!

How OTP's work

Here's a list of steps that describes the whole One Time Password protocol (shown above, second image), from start to end.

1. The User-Agent POSTs: { email: "chris@example.com" } to /auth/v2/request on the app server (lets use webmaker.org)
2. The Webmaker.org server should forward the post body to the Basic Auth protected route /auth/v2/request on login.webmaker.org
3. The login server will look up the user account by the provided email address. If none is found, responds to the request with a 400: "User not found"
4. A one time password is generated using Hat.
5. The password is 24 random bits and is converted to base 36. This generates a five character string of letters and/or digits.
6. The password is set to expire thirty minutes after creation, and is passed to Webmaker's event processing queue Sawmill. Its powered by Amazon's Simple Queue Service (SQS), and messages are sent using Hatchet, which uses Amazon's aws-sdk module.
7. From Sawmill, The event is converted into an email using a template, and is forwarded to lumberyard
8. Lumberyard sends the email to the one stored in User account the OTP was generated for. This uses the aws-sdk and Amazon's Simple Email Service (SES)
9. Once the User gives the User-Agent the OTP it POSTs: { email: "chris@example.com", token: "s34xa" } to /auth/v2/authenticateToken on the app server
10. The Server will forward the post body to /auth/v2/authenticateToken on login.webmaker.org
11. The login server will attempt to fetch the user and login token from the database, ensuring that the fetched token was 1. created no more that thirty minutes from the system time on the server and 2. has not been used to log in already.
12. Should the criteria not be met, a 401 reponse it returned to the app server, who should not issue a session cookie.
13. If all criteria is met, the token is marked used and saved, and the user account object is serialized into session format and returned to the app server.
14. The app server should then send a SET-COOKIE header to the User-Agent. It should be set to https only and be encrypted with a session secret.

What Now?

With the code in a working state, it's just a matter of iteration. Code review, fixes + optimizations, repeat. We also have folks working on copy for the emails and modal dialogs. One thing that I think is going to be the most challenging for us when we roll out the new system is clearly communicating to users the new changes. Whether it be through email comms, banners on the homepage or a first-run log-in experience, I feel it's incredibly important to get this right.

We've been thinking about keeping Persona as an optional log-in method (+1) and implementing an opt-in, run-of-the-mill password login system (I have implemented this fully, but it was cut from this iteration D:). I've also experimented (successfully) with turning login.webmaker.org into an OAuth2 provider - but havent actually gotten any resources or scopes set up to work with the oauth tokens I can generate.

I've also reached out to the security gurus in the organization, with the goal of going over the new protocol with them and getting feedback on the right and the wrong.

That said, what do you think? Have you had experience with log-in flow like this? What challenges did you face or forsee this flow facing? Feel free to use the comment section below, drop some comments in the Bugzilla bug or email me

Webmaker is a very large project, with dozens of parts that all come together at https://webmaker.org. One of the hardest things for new contributors is getting everything set up properly. This problem multiplies ten-fold when the desired platform of the developer is Windows. Recently, I put together a guide for a new contributor, explaining the steps it takes to set up a Windows 7 computer for development of Webmaker projects (specifically, the events platform). It's a long, complicated, and frustrating process, which I'd like to share here, so that should anyone need to do this, they won't have to look very far.

Dependency Installation

Git

We use Git for version control, and Github for hosting our code repos publicly.

You can install Git from http://git-scm.com/download/win. That site should automatically prompt you to download the latest version of Git available for Windows. Once it is downloaded, run the executable to begin the installation. I used all the default settings.

Node

Basically all of Webmakers servers are written in JavaScript, and use NodeJS.

Download the latest installation executable from http://nodejs.org/download and run the installer once the download completes. Node Package Manager (npm) will be installed as well.

Python

Some modules included as dependencies will require Python 2.7 to be installed.

You can download the installer from https://www.python.org/download/releases/2.7.8/. During the installation make sure to specify that you want python.exe added to your PATH.

Visual Studio 2010

This is where things get stupid. Npm depends on something called node-gyp, a so-called "cross-platform" tool for compiling native addons. The problem is, its a pain in the ass to get it running on a Windows machine. I'll share what I did to get it running, but in the end, you might have to do something different, depending on your machine.

• First, I installed Visual C++ 2010 Express from microsoft.com.

• I then installed the Windows SDK 7.1

• Next up was SP1 for Visual Studio 2010

• Lastly, I installed the Visual C++ 2010 SP1 Compiler Update for the Windows SDK 7.1

Installing the four items above didn't quite cut it for me though. Here's the last few steps I took:

• I ran git bash (you can find it in your start menu) and ran the command npm install -g node-gyp to install the node-gyp package globally.

• I then opened the file C:\Users\cade\.node-gyp\0.10.29 \common.gypi - obviously, substitute your username and the version of node-gyp you install.

• The file contains configuration settings in JSON format. Navigate to target_defaults.msvs_settings.VCLinkerTool and add 'AdditionalLibraryDirectories': 'c:\\Program Files\\Microsoft SDKs\\Windows\\v7.1\\Lib\\x64', to the object VCLinkerTool

If you're as lucky as I was, node-gyp might now work for you!

Grunt-cli and bower

Grunt is a task runner, used maily for building resources. Bower is a front-end package manager. We're going to install these globally using npm, so that the grunt and bower commands will be added to your PATH, making life easier. To do this run: npm install -g grunt-cli bower

Setting up Webmaker Events

Login.webmaker.org

This is the Login server for webmaker. It manages Webmaker accounts through the Login API, and is required for log-in functionality to work. The code can be found at https://github.com/mozilla/login.webmaker.org

1. In Git Bash, change into the directory you would like to clone the code into.

2. Run git clone https://github.com/your_user_name/login.webmaker.org to clone your fork of the code to your machine. Git will set your Github fork up as the remote named "origin", so you can push code changes and branches back up to the website.

3. Change into the login.webmaker.org folder and run cp env.sample .env. This will create a configuration file for the server from the default one provided in the repo. It's configured to work "out of the box" with other Webmaker sites and services you will run locally.

4. Now we need to install npm and bower dependencies. Do this by running npm install; bower install

5. You should now be able to start the server with node app. If Windows prompts you to grant permissions for Node, accept them.

Events Front End

This is the Front-End for The Webmaker Events platform. It's an Angular app that is served using Express. The source code is at https://github.com/mozilla/webmaker-events-2. Don't ask about webmaker-events, we don't talk about it anymore.

1. In Git Bash, change into the directory you would like to clone the code into.

2. Run git clone https://github.com/your_user_name/webmaker-events-2 to clone your fork of the code to your machine. Git will set your Github fork up as the remote named "origin", so you can push code changes and branches back up to the website.

3. Change into the webmaker-events-2 folder and run cp .env-dist .env. This will create a configuration file for the server from the default one provided in the repo. It's configured to work "out of the box" with other Webmaker sites and services you will run locally.

4. Now we need to install npm and bower dependencies. Do this by running npm install; bower install

5. Before we can run the server we need to compile the CSS. Do so by running grunt build

6. You should now be able to start the server with node server/server.js. If Windows prompts you to grant permissions for Node, accept them.

Events Service

This is the Events API, which provides RESTful read/write access to the events database. The source code can be found at https://github.com/mozilla/webmaker-events-service.

1. In Git Bash, change into the directory you would like to clone the code into.

2. Run git clone https://github.com/your_user_name/webmaker-events-service to clone your fork of the code to your machine. Git will set your Github fork up as the remote named "origin", so you can push code changes and branches back up to the website.

3. Change into the webmaker-events-service folder and run cp .env-dist .env. This will create a configuration file for the server from the default one provided in the repo. It's configured to work "out of the box" with other Webmaker sites and services you will run locally.

4. Now we need to install the npm dependencies. Do this by running npm install

5. You should now be able to start the server with node server/server.js. If Windows prompts you to grant permissions for Node, accept them.

Summary

Developing on windows with NPM sucks, but it's not impossible. If the above helped, I'm glad. The steps for the above set-ups for login, events and the events-service can be applied to most other Webmaker sites and tools. Be sure to always check the README files for details about set-up and running a particular project.

I somehow managed to break my blog...

Update: Blog online, data missing. I'm currently working on recovering it.

update: As of 10:15 PM EDT, the blog is back online!

What happened? Well, It all started when I decided it would be a great idea to update my Ghost installation to 0.4.2. I'd done an upgrade before without much trouble so I thought everything would be okay. Unfortunately, Ghost wasn't on board with my plan.

I started out by checking out the latest version of ghost on my server, which went smoothly. Next, I ran sudo sh -c "npm install; bower install; grunt; grunt prod" to get all the dependencies and build all the assets. This went off without a hitch. I restarted my Ghost service and verified it was running.

This is where things get strange. My blog's content seemed to be just fine. But when I attempted to log into the admin console, it would reject the username and password for my account. I was a little annoyed, but not too concerned. I figured I could just reset the password. Nope. Upon trying to send a reset email, I was given an error message claiming that the Gmail account I configured for reset emails couldn't be authenticated. Thats when it hit me: The account was a dummy account I created with the sole purpose of sendign me password reset emails. The very same dummy account that I had gotten an email from Google about one week prior stating it was being closed due to TOS violations. -.-

A simple fix really. Just have to update those settings! Since I have a nice new AWS account with a bunch of credit to burn through, I figured I'd hook it up to SES. I logged into my console, and generated the creds to send emails using SES. I plugged them in, rebooted the service, fired up the forgotten password page and.... Nope! More errors. This time something about bad credentials. AWS is great, but it can be incredibly picky about permissions. After some more fiddling and prodding though, I did manage to get password reset emails working.

I clicked the link in the email from my blog, and it brought me to the password reset page. I filled in a new password and hit submit. Navigating back to the login screen, I proceeded to enter in the new login and password. Upon hitting submit, I was greeted with a lovely error about it not being the right username/password ......WAT?!?

Something's not right, I thought. I'm going to run the service manually and take a peek at the output. What I saw was pretty disturbing (for me at least). Something along the lines of "Your database is incompatible with this version of Ghost - You will have to create a new one". No problem, I though, I'll just switch back to the old version!

I checked out the previous version I was running, ran the scripts to set everything up and booted up the service manually. I loaded my home page, which displayed perfectly. I then tried to get to the login screen. This is when I really started to sweat... It wouldn't load! I would wait for a dozen seconds or so, and nginx (used for proxying) would 502 me! Looking in the console, I saw a message along the lines of "Unhandled exception: session table doesn't exist". I had no idea what was up.

Things didn't look good for chrisdecairos.ca... I decided my only option was to copy my contents directory (contains the sqlite db file, images, and themes directory) and my configuration file somewhere safe so that I could wipe the slate clean on my ghost installation. I downloaded the official release zip of ghost 0.4.2 and replaced my installation with its contents. I double checked that all the dependencies were in plave and that all assets were built. I copied over my config file, and started up the service.

It worked. I was able to log into a new account and regain access to the admin tools. I re-instated all the themes settings to get the look and feel of the blog back. With everything back up and running, the only thing left for me was to migrate all my posts and tags into the new db file. With a bit of research, I found what I needed:

sqlite3 ghost.db .dump > ghost.sql

Which dumps the sql to create your DB into a file. I opened up that file and deleted all the statements that weren't related to posts, tags and posts_tags. I also removed the create sytax for those tables. I should mention that I compared the schemas between the old db file and the new, and they were identical. This meant that once I ran the script, I should be back in business. Anyways, I scp'ed the script to my server, logged in and ran:

sqlite3 content/data/ghost.db < ~/ghost.sql

and crossed my fingers.....

When I reloaded my blog, everything was back to normal! I had succeeded! I had also lost most of my evening -.-

The takeaways from this are two:

1. Back up your database, you dummy!
2. Git checkout is a horrible way to update blog software.

But hey, at least I know how to recover my blog in situations like this.

Since the creation of Webmaker.org and the MakeAPI, the recommended strategy for creating and curating galleries of makes was to use tagging.

For example, one can tag ten makes with gallery and surface that collection of makes using a tag search. If there was a need for order in the data, a second unique tag must be given to each make as an indication of its order in the set. For example: gallery-1 all the way to gallery-10, then they must be sorted manually after fetching the set.

The downsides of this for the regular user are numerous:

1. If someone else uses the same tag on an unrelated make, it will "pollute" the gallery
2. The logic required to order the gallery is needlessly complex.
3. It's a pain in the ass to apply tags to a set of existing makes
4. It's a pain in the ass to update a gallery without screwing something up horribly (i.e. duplicate ordering tags)

Galleries and collections are a heavily used feature across the Webmaker universe, and the horrible system we force people to use to build these galleries is a crime.

In order to rectify this crime against the world, I identified some goals that would have to be met in order for this to be considered fixed.

1. A gallery should not be defined by tags
2. Gallery order should not be defined by tags, and should not require additional logic at the point of the consumer to sort the gallery data
3. Fetching Make data from a gallery should be simple, and act as a drop in replacement old style of gallery searches.
4. Every Webmaker User should be able to create and maintain their own galleries of Makes

Enter the Make List API. It is a collection of server endpoints that allow Webmaker apps to manage a newly defined data model built right into the Make API server.

Essentially, a Make List ("List") is a Mongoose model that stores an Array of Make ID's.

Goal 1 Completed!

Order in the array determines positioning wherever the List is to consumed.

When a List is fetched, The Make API will automatically fetch the Makes from ElasticSearch, sort them, and return the Make JSON to the client. The data is identical to the data returned by make searches.

Goal 2 and 3 Completed!

Lastly, every list is associated with a Webmaker account and only an owner of an List can modify it.

Goal 4 completed!

So the TL;DR; of this is that any user can create and curate any number of Lists (my favourites, top tens, best of makerparty, etc). These lists can be built into any number of places - The front page, the webmaker gallery, top webmaker teaching kits, webmaker profile, webmaker events, ALL THE THINGS!

If I had this running somewhere live I would link it, but I do not. For the time being, you can follow the development in Bug 997329 on Bugzilla.

Below is a Youtube Video of the Demo Application I built to show off the API:

If you have cool ideas about what this can be used for, or have questions about if it could be used for something, drop me a line!

I'm going to be a dad!!!

I'm proud to announce that Sarah and I will be welcoming a new member into our family this November!!