Altering large tables without bringing down your service

When we run ALTER statements on our big tables we have to plan ahead to keep from breaking whatever service is using the database. In MySQL, many times* a simple change to a column (say, from being a short varchar to being a text field) can read-lock the entire table for however long it takes to make the change. If you have a service using the table when you begin the query you'll start eating into your downtime budget.

If you have a large enough site to have database slaves you'll have a double-whammy - all reads will block on the master altering the table, and then, by default, the change will be replicated out to your slaves and not only will they read-lock the table while they alter it, but they will pause any further replication until the change is done, potentially adding many more hours of outdated data being returned to your service as the replication catches up.

The good news is, in some situations, we can take advantage of having database slaves to keep the site at 100% uptime while we make time consuming changes to the table structure. The notes below assume a single master with multiple independent slaves (meaning, the slaves aren't replicating to each other).

Firstly, it should go without saying, but the client application needs to gracefully handle both the existing structure and the anticipated structure.

When you're ready to begin, pull a slave out of rotation and run your alter statement on it. When it completes, put the slave back into the cluster and let it catch up on replication. Repeat those steps for each slave. Then failover one of the slaves as a new master and pull the old master out of rotation and run the alter statement on it. Once it has finished put it back in the cluster as a slave. When the replication catches up you can promote it back to the master and switch the temporary master back to a slave.

At this point you should have the modified table structure everywhere and be back to your original cluster configuration.

Special thanks to Sheeri who explained how to do all the above and saved us from temporarily incapacitating our service.

*What changes will lock a table vary depending on the version of MySQL. Look for "Allows concurrent DML?" in the table on this manual page.

Contributing a patch to the Firefox Marketplace from scratch

Jared, Stuart, and Andy recently spent some time focusing on one of the Marketplace's biggest hurdles for new contributors: how do I get all these moving pieces set up and talking to each other?

I haven't written a patch for the Marketplace in a while so I decided to see what all the fuss was about. First up I, of course, read the installation documentation. Ok, I skimmed it, but it looks pretty straight forward.

Step 1: Install Docker

I'm running Ubuntu so that's as easy as:

sudo apt-get install

To fix permissions (put your own username instead of clouserw):

sudo usermod -a -G docker clouserw

Step 2: Build Dependencies

The steps below had lots of output which I'll skip pasting here, but there were no errors and it only took a few minutes to run.

$ git clone
$ cd wharfie

$ bin/mkt whoami clouserw
$ bin/mkt checkout

$ mkvirtualenv wharfie
$ pip install --upgrade pip
$ pip install -r requirements.txt
$ sudo sh -c "echo >> /etc/hosts"

Step 3: Build and run the Docker containers

I ran this seemingly innocent command:

$ fig build

And 9 minutes and a zillion pages of output later I saw a promising message saying it had successfully built. One more command:

$ fig up

and I loaded in my browser:

Screenshot of Marketplace

A weird empty home page, but it's a running version of the Marketplace on my local computer! Success! Although, I'm not sure it counts unless the unit tests pass. Let's see...

$ CTRL-C  # To shutdown the running fig instance
$ fig run --rm zamboni python ./ test --noinput -s --logging-clear-handlers
Ran 4223 tests in 955.328s
FAILED (SKIP=26, errors=34, failures=17)

Hmm...that doesn't seem good. Apparently there is some work left to get the tests to pass. I'll file bug 1082183 and keep moving. I know Travis-CI will automatically run all the tests on any pull request so any changes I make will still be tested -- depending on the changes you make this might be enough.

Step 4: Let's write some code

If I were new to the Marketplace I'd look at the Contributing docs and follow the links there to find a bug to work on. However, I know Bug 989121 - Upgrade django-csp has been assigned to me for six months so I'm going to do that.

I'll avoid talking about the patch since I'm trying to focus on the how and not the what in this post. The code is all in the /trees/ subdirectory under wharfie, so I'll go there to write my code. A summary of the commands:

$ cd trees/zamboni
$ vi <files>  # Be sure to include unit tests
$ git add <files>
$ git checkout -b 989121  # I name my branches after my bug numbers
$ git commit  # commit messages must include the bug number
$ git push origin 989121

Now my changes are on Github! When I load the repository I committed to in my browser I see a big green button at the top asking if I want to make a pull request.

Github pull request button

I click the button and submit my pull request which notifies the Marketplace developers that I'd like to merge the changes in. It will also trigger the unit tests and notify me via email if any of them fail. Assuming everything passes then I'm all done.

This flow is still a little rough around the edges, but for an adventurous contributor it's certainly possible. It looks like Bug 1011198 - Get a turn-key marketplace dev-env up and running is tracking progress on making this easier so if you're interested in contributing feel free to follow along and jump in when you're comfortable.

Migrating off Wordpress

This post is a celebration of finishing a migration off of Wordpress for this site and on to flat files, built by Jekyll from Markdown files. I'm definitely looking forward to writing more Markdown and fewer HTML tags.

90% of the work was done by jekyll-import to roughly pull my wordpress data into Markdown files, and then I spent a late night with Vim macros and sed to massage it the way I wanted it.

If all I wanted to do was have my posts up, I'd be done, but having the option to accept comments is important to me and I wasn't comfortable using a platform like Disqus because I didn't want to force people to use a 3rd party.

Since my posts only average one or two comments I ended up using a slightly modified jekyll-static-comments to simply put a form on the page and email me any comments (effectively, a moderation queue). If it's not spam, it's easy to create a .comment file and it will appear on the site.

My original goal was to host this all on Github but they only allow a short list of plugins and the commenting system isn't on there so I'll stick with my own host for now.

Please let me know if you see anything broken.

The Great Add-on Bug Triage

The AMO team is meeting this week to discuss road maps and strategies and among the topics is our backlog of open bugs. Since mid-2011 there has averaged around 1200 bugs open at any one time.

Currently any interaction with AMO's bugs is too time consuming: finding good first bugs, triaging existing bugs, organizing a chunk of bugs to fix in a milestone -- they all require interacting with a list of 1200 bugs, many of which are years old and full of discussions by people who no longer contribute to the bugs. The small chunks of time I (and others) get to work on AMO are consumed by digging through these old bugs and trying to apply them to the current site.

In an effort to get this list to a manageable size the AMO team is aggressively triaging and closing bugs this week, hopefully ending the week with a realistic list of items we can hope to accomplish. With that list in hand we can prioritize the bugs, divide them into milestones, and begin to lobby for developer time.

Many of the bugs which are being closed are good ideas and we'd like to fix them, but we simply need to be realistic about what we can actually do with the resources we have. If you contribute patches to any of the bugs, please feel free to reopen them.

Thanks for being a part of AMO.

Retiring AMO's Landfill

A few years ago we deployed a landfill for AMO - a place where people could play around with the site without affecting our production install and developers could easily get some data to import into their local development environment.

I think the idea was sound (it was modeled after Bugzilla's landfill) and it was moderately successful but it never grew like I had hoped and even after being available for so long, it had very little usable data in it. It could help you get a site running, but if you wanted to, say, test pagination you'd still need to create enough objects to actually have more than one page.

A broader and more scalable alternative is a script which can make as many or as few objects in the database as you'd like. Want 500 apps? No problem. Want 10000 apps in one category so you can test how categories scale? Sure. Want those 10000 apps owned by a single user? That should be as easy as running a command line script.

That's the theory anyway. The idea is being explored on the wiki for the Marketplace (and AMO developers are also in the discussion). If you're interested in being involved or seeing our progress watch that wiki page or join #marketplace on