Kumar leapfrogged that milestone by adding Vagrant configuration scripts to our repository. Now you can have a running version of AMO on your local system in about 3-5 commands[1].

Check out the steps to install AMO with Vagrant to see how. I set it up on OS X last week and aside from waiting for the download it only took a few minutes. Drop by #amo on IRC if you run into any troubles.

[1] Setting this up on Windows is apparently more difficult although one contributor did find success after fighting with it for some time.

Edit: Updated the installation instructions URL

I’ve customized irssi with custom highlights and commands and all has been well, save one thing: bug 310 – vertical splits. Irssi can’t do vertical window splits and with the trends giving us widescreen monitors, horizontal splits aren’t really useful to me. The bug has been open since 2005 and I’ve all but given up on it.

Then the other day Chris McDonald claimed the unfortunately named WeeChat was superior to Irssi. I was all ready to defend Irssi’s honor but I read through the documentation and WeeChat was pretty compelling – most notably its support for vertical splits!

So, long story short, I switched last week and it’s awesome. I don’t miss Irssi at all and in fact WeeChat offers me things I never even knew I’d want (like per-buffer history when hitting the up-arrow). I’ve customized it so it essentially looks like Irssi (no nickname list, etc.) and I’ve added a lot of aliases to make it easier for me to use (mostly vim keys). If you’re an Irssi user you owe it to yourself to at least read their docs and see what you think.

A big thanks to all the developers who made the switch possible, and especially the ones at the end who were working on migrating PHP scripts instead of more glorious projects. Thanks to the management and all the other folks affected by the switch for being patient with the scheduling. Thanks to the localizers for dealing with the crazy merged .po files, and, of course, thanks to the users of AMO for reporting bugs when they happened and generally being an all around great community to work with.

[1] Currently all traffic is being rewritten to a WSGI handler for Python. PHP is still on the server but nothing uses it. We’ll be removing it completely in the near future.

Last March there was some feedback from developers about being tired of the code they were working on and eyeing other technologies but not having the time to work with them. This is a fragile balance which every manager and developer has had to struggle with. Writing new code is always sexier than maintaining old code, but the old code is the bread and butter that keeps you in business. Finding a happy medium is a noble, elusive, and unfortunately, shifting goal.

Enjoying what you work on is crucial to being happy and productive though so I’ll continue to pursue that goal.

One of the changes we experimented with at the time was to let developers choose their own focus for the quarter. We still had high level quarterly goals that needed to be done, but that left plenty of other time throughout the quarter to work on any of those areas which were all equally important but perhaps not equally as interesting. I filled up a whiteboard with ideas one morning (and left space for developers to add more) and we had a meeting later in the day where anyone could discuss the ideas and sign up for what was interesting to them. This was a short term commitment from the developers and they got to work on what they cared about. From a managerial perspective, it increased motivation but didn’t sacrifice accountability. Since I knew the focus of the developers early in the quarter, I could help clear out roadblocks that they’d meet before they even started working on their areas.

The quarter is over this week and overall the idea was a success. All the feedback from developers was positive – if you’re looking for a way to spice up the top-down approach to goal setting this was effective. It also turned the goal setting into more of a discussion rather than an edict that just shows up.

I wrote this almost 9 months ago but apparently never hit publish. So, here it is. We’ve played with it a bit since but haven’t had a large quarterly planning meeting like that because we’re mainly focused on the marketplace. Experimenting with smaller goals is a next step here, and doing the meetings on a weekly or biweekly basis.

I’m sorry, but there’s not enough air in here for everyone. I’ll tell them you were a hero.

I had an idea to translate this idea into photographs and tell a story within a limit of 3 photos (a generous 3000 words if we’re going by the standard exchange). I took the first two photos fairly quickly but the 3rd took me a long time to organize the scene (and get a participant). During some time off a couple weeks ago I found the time to finish the trio of photos and complete the tale.

Click on the photo below to see the whole story

SSL Encryption: Let’s start off easy. Anytime you go to addons.mozilla.org we redirect you to https://addons.mozilla.org. Assuming you make it through the redirect safely you can be reasonably sure you’re talking to us at that point. Any data sent between your browser and us is encrypted with industry standard encryption. This seems like a freebie (I know you’re thinking, “really? you’re talking about SSL?”), but do a quick search and you’ll find plenty of financial institutions that fail to take even this most basic precaution on pages where you submit the username and password to your accounts.

Alright, let’s get more interesting. AMO has a lot of user uploaded data on it, from images to files (the add-ons themselves) to the files within add-ons (we allow you to browse uploaded add-ons on the site). If a user uploads some malicious JavaScript they’ll be able to run it in the context of the addons.mozilla.org domain which would give them access to manipulate the site and change or steal user data. We protect ourselves by using an alternate domain for user uploads – static.addons.mozilla.net. By using .net instead of .org we’ve sand-boxed user scripts onto their own domain and protected the content on .org. This is industry standard (notice how your content you upload to Google comes off of www.googleusercontent.com) and gives you a free performance boost as well.

Actually, uploaded images should get special mention. It’s a best practice to always clean and verify user data but this is often overlooked for images. Back in the days of IE6 you could actually run arbitrary JavaScript embedded in the comments of an image. This has since been fixed in the browser but poorly configured servers and applications can still pose a threat. Neal Poole demonstrated a proof of concept on a Mozilla site where he embedded PHP in an image, saved it as “image.php” and uploaded to a site. The site saved it under a /media/gallery/ directory (under the webroot with PHP enabled) and he had arbitrary PHP execution on the server. The lesson learned was always re-encode user images when they are submitted. Even if you’re re-encoding from PNG to PNG, strip the comments – it’s not worth it to find out there was something malicious in them later on.

For many sites session cookies are one of the most valuable assets behind the actual credentials to log in. AMO protects the session cookie (and most of its cookies) with two very underused options: the Secure and HTTPOnly flags. Secure simply means the cookie is only sent over a secure connection – that means that when you go to addons.mozilla.org without typing the https, your cookies (and therefore your session) aren’t sent and won’t be compromised if someone is eavesdropping. HTTPOnly means that the cookies are sent with browser traffic to AMO but the cookie is inaccessible to client side scripts. If a malicious script is somehow injected into the page this option will prevent it from stealing the session id. Assuming you don’t need access to the cookies and are running SSL these are essentially free additional layers of security for your site.

Every request to AMO returns a pile of interesting (and sometimes bleeding edge) HTTP headers. If you hit the front page, you’ll see X-Frame-Options: DENY. In a supporting browser, this will prevent someone from putting the AMO site into an <iframe> (which prevents things like clickjacking). The vast majority of sites can add this header for more free security.

In a couple examples above I say that once you get to AMO on SSL you’ll be fine but I conveniently skip all the traffic and redirects up until then. An attempt to keep people safe until they reach that point is the Strict-Transport-Security: max-age=2592000 header. This tells the browser that for the next 30 days, if you type in addons.mozilla.org without https it will automatically send it over SSL before the initial request – no unencrypted traffic at all. Support for this header is not widespread yet, but it’s in all the recent versions of Firefox and I expect support for it to expand.

I can’t mention “bleeding edge” and “headers” without a hat tip to the Content Security Policy (specifications). We’ve had it in reporting mode for a couple months as we work out what needs to be adjusted before we turn it on, but once we do, this will (again, in a supporting browser) define specific rules for what domains will have valid assets (like images, JavaScript, CSS, etc.) as well as disallow any inline JavaScript from executing. This essentially locks down XSS attacks even if someone does find a way to inject code into the page. It’s a really exciting development but not for the faint of heart to implement on a complex site at this point in time. There are still some bugs to iron out and some edge cases to clarify in CSP but it’s becoming something to seriously consider. I think Twitter is the most prominent site using it to enforce rules (as opposed to only reporting violations) at this time.

Whew, that is a pile of text and that’s just covering the extreme front end. I’m going to cut it off there to keep this from running on for pages but if there is interest I’ll write another post about more things AMO does to defend itself and its visitors, and other areas where everyone can consider adding in extra security.

In the mean time, if you want even more best practices the Mozilla Security Team has made a great wiki page for further reading about web security.

It turned out pretty well so here is addons.mozilla.org development so far for 2011 (in HD!):

(Warning: prefetching is off but if you click play you’re in for an 80MB video.)

The gource docs are easy to read if you want to do this for your project, but for the record this is what I ran:

gource --viewport 1280x720 \
--user-image-dir ~/sandbox/zamboni/.git/avatar/ \
--title "addons.mozilla.org" \
--auto-skip-seconds 1 \
--seconds-per-day 1 \
--start-position .715 \
--max-file-lag 0.5 \
--max-files 5000 \
--camera-mode track \
-o -


Piped to:

ffmpeg -y \
-r 60 \
-f image2pipe \
-vcodec ppm \
-i - \
-vcodec libtheora \
-b 10000K \
~/out.ogv


That gave me a 180MB uncompressed ogv. The uncompressed version looks far nicer, but that’s a lot of bandwidth for a random video so I cut it down with ffmpeg2thoerea (anyone know the switch to do this directly in the ffmpeg command?):

ffmpeg2theora -v 4
~/out.ogv
--optimize
--noaudio
-o amo-2011-development.ogv


As is all too common, way leads on to way, and now here we are three years later. getpersonas.com has become a juggernaut of 3000x200px free expression on the web. There are over 1.25 million registered users on the site, 400,000 personas, and a half million hits a day. The site was built with scaling in mind and, honestly, has needed relatively little attention.

On the other hand, the site lost its owners and maintainers last year. Deb stepped up with some awesome volunteers and contractors to fix minor issues but there are no dedicated developers to keep the site fresh. The web security bounty program late last year wasn’t kind to the old code, and any time devoted to the site turned in to trudging through old PHP code to solve overlooked problems from long ago.

We’ve decided that this is the year to finally replace the precarious cron job synchronizing the getpersonas.com and AMO databases for the past 18 months and finally migrate the site to AMO completely. This is no small undertaking, but we’ve had a lot of time to think about it.

I wrote a migration plan a few weeks ago as a general guide. The searching and listing pages are already at parity with getpersonas.com. The reviewer and author functionality will be added shortly – and if you read the bugs and look at the mockups you’ll see it’s greatly improved. This is a mutually beneficial migration; the personas will be able to leverage AMO features like statistics reporting and collections, and AMO will get a fresh look at reviewing user submitted content and an influx of creative designers.

I snuck in to a personas planning meeting last week and I saw a lot of fun stuff in the pipeline for personas. I’m happy to say migrating them onto AMO will give everyone the server and developer resources to get that new stuff out the door. This will get underway in Q3 of this year.

With the launch of landfill.amo[1] we bypass the entire headache. The site started with a clean database and I uploaded an add-on to show it worked, but otherwise it’s empty. It’s compact, fast, and simple to use. The beauty of the site for volunteers and casual developers is that the database and the filesystem are available in their entirety to download. This means you can check out the code, fill in the configuration, import the landfill database and have the site 90% running.[2]

Perhaps a testament to the obscene number of open bugs for AMO right now, but this also solves a second long standing problem where localizers couldn’t see the entire site. On landfill, anyone can be an administrator, an editor, or any other permission level they’d like; and they’ll be able to see the entire site.

If you’ve been overwhelmed or frustrated trying to set up AMO in the past, now is a good time to give it another shot. The landfill should just get better with age and use – if a few people register and add some data the available database dumps will get richer.

If there is a part of the site that isn’t working and you need it to be, let me know. Keep in mind this is only the new Python code, so the few parts that are still on PHP (like the admin panel) won’t be available until they are ported. Code is updated near-instantly on commit, localization changes are updated every 5 minutes.

[1] Forgive the fake certificate. This is a sandbox for developers, y’all know what you’re doing.

[2] Honestly, 90% is really all you need. We do a lot of stuff for scalability, statistics, etc. and unless you’re actually working on that part of the site, you don’t need those elements running. Of course, you’re more than welcome to turn them on, I’m just trying to make it easy.

In January of 2010 we started migrating addons.mozilla.org from CakePHP to Django. It was a controversial decision. Developers were ambivalent to excited, managers were opposed to neutral – a split anyone would expect. When I first talked about it I expected to be able to turn off PHP by the end of the year. It didn’t turn out quite like that.

Fifteen months later we’re still transitioning and it’s still stressful. The toughest part about a major migration like this is that there is only one team that is doing the migration, continuing to add the new features we need, and all the while maintaining the old site. That’s a stressful environment for developers since the interactions between the languages can be complicated, it’s stressful for managers because features take longer to complete, and it’s stressful for users (and QA for that matter) because issues will arise which are hard to reproduce and complicated to explain.

In the midst of all the work of migration, the rest of the company is still working: the security team is announcing bounties on our site which means we need to be vigilant about fixing issues, project management continues to come up with features to be added, the site perseveres in its never-ending quest for a new look and feel, and Firefox 4 is using AMO like never before meaning approaching 10,000 hits per second is a regular day. All of that is specific to the add-ons site, but consider your own company if you’re thinking of going down the same road – what is coming up for your site that will throw a wrench in the works?

The meat and potatoes of it really comes down to: Given the hindsight of today, would the migration be a good idea? There isn’t a right answer for every site, but for AMO we did the right thing[1]. As of today the majority of pages that matter are on Python – there are some admin tools, and some cron jobs, and the occasional semi-obsolete public page that is on PHP, but for the most part, we’re looking really good (less hand waving, more real data). My new (overly optimistic?) plan is to have PHP off by the end of this year. We’ll see.

To give you an idea of man-hours, we’ve had anywhere from 3 to 6 superhero developers working on the site over the past 15 months, and it’s looking like the whole thing will take around 24 months. That’s a big chunk of time for a site that needs to grow and evolve as quickly as popular sites do these days.

So, overall, I think the lesson is: any reasonably sized site is going to have rabbit holes in it. At first glance AMO might look like it’s got a dozen “main” pages, with a couple dozen more supporting pages (and throw in a few more for the admin CRUD). Have a look at that spreadsheet I linked above and you’ll see that’s not even remotely the case. The spreadsheet even ignores sub-pages in a few places and doesn’t include any new features added in the past year. If you’re considering a migration, think it through well. Make a spreadsheet of every URL, identify the complicated areas, and make sure everyone is clear on the timeline and what it means for new features. People will absolutely try to scope creep your migration – make it clear if a section of the site is migrating as-is or can be migrated and redesigned at the same time. Redesigns add complexity for the developers but can earn you some good will with the users and managers and if you’re in this boat you can use all the good will you can get.

May you have the best of luck with your decisions.

[1] I’ll write another post about pros/cons of the actual frameworks and platforms. Let’s just assume we’re happy with the technical side of the switch for now.