Kumar leapfrogged that milestone by adding Vagrant configuration scripts to our repository. Now you can have a running version of AMO on your local system in about 3-5 commands[1].

Check out the steps to install AMO with Vagrant to see how. I set it up on OS X last week and aside from waiting for the download it only took a few minutes. Drop by #amo on IRC if you run into any troubles.

[1] Setting this up on Windows is apparently more difficult although one contributor did find success after fighting with it for some time.

Edit: Updated the installation instructions URL

I’ve customized irssi with custom highlights and commands and all has been well, save one thing: bug 310 – vertical splits. Irssi can’t do vertical window splits and with the trends giving us widescreen monitors, horizontal splits aren’t really useful to me. The bug has been open since 2005 and I’ve all but given up on it.

Then the other day Chris McDonald claimed the unfortunately named WeeChat was superior to Irssi. I was all ready to defend Irssi’s honor but I read through the documentation and WeeChat was pretty compelling – most notably its support for vertical splits!

So, long story short, I switched last week and it’s awesome. I don’t miss Irssi at all and in fact WeeChat offers me things I never even knew I’d want (like per-buffer history when hitting the up-arrow). I’ve customized it so it essentially looks like Irssi (no nickname list, etc.) and I’ve added a lot of aliases to make it easier for me to use (mostly vim keys). If you’re an Irssi user you owe it to yourself to at least read their docs and see what you think.

A big thanks to all the developers who made the switch possible, and especially the ones at the end who were working on migrating PHP scripts instead of more glorious projects. Thanks to the management and all the other folks affected by the switch for being patient with the scheduling. Thanks to the localizers for dealing with the crazy merged .po files, and, of course, thanks to the users of AMO for reporting bugs when they happened and generally being an all around great community to work with.

[1] Currently all traffic is being rewritten to a WSGI handler for Python. PHP is still on the server but nothing uses it. We’ll be removing it completely in the near future.

It turned out pretty well so here is addons.mozilla.org development so far for 2011 (in HD!):

(Warning: prefetching is off but if you click play you’re in for an 80MB video.)

The gource docs are easy to read if you want to do this for your project, but for the record this is what I ran:

gource --viewport 1280x720 \
--user-image-dir ~/sandbox/zamboni/.git/avatar/ \
--title "addons.mozilla.org" \
--auto-skip-seconds 1 \
--seconds-per-day 1 \
--start-position .715 \
--max-file-lag 0.5 \
--max-files 5000 \
--camera-mode track \
-o -


Piped to:

ffmpeg -y \
-r 60 \
-f image2pipe \
-vcodec ppm \
-i - \
-vcodec libtheora \
-b 10000K \
~/out.ogv


That gave me a 180MB uncompressed ogv. The uncompressed version looks far nicer, but that’s a lot of bandwidth for a random video so I cut it down with ffmpeg2thoerea (anyone know the switch to do this directly in the ffmpeg command?):

ffmpeg2theora -v 4
~/out.ogv
--optimize
--noaudio
-o amo-2011-development.ogv


As is all too common, way leads on to way, and now here we are three years later. getpersonas.com has become a juggernaut of 3000x200px free expression on the web. There are over 1.25 million registered users on the site, 400,000 personas, and a half million hits a day. The site was built with scaling in mind and, honestly, has needed relatively little attention.

On the other hand, the site lost its owners and maintainers last year. Deb stepped up with some awesome volunteers and contractors to fix minor issues but there are no dedicated developers to keep the site fresh. The web security bounty program late last year wasn’t kind to the old code, and any time devoted to the site turned in to trudging through old PHP code to solve overlooked problems from long ago.

We’ve decided that this is the year to finally replace the precarious cron job synchronizing the getpersonas.com and AMO databases for the past 18 months and finally migrate the site to AMO completely. This is no small undertaking, but we’ve had a lot of time to think about it.

I wrote a migration plan a few weeks ago as a general guide. The searching and listing pages are already at parity with getpersonas.com. The reviewer and author functionality will be added shortly – and if you read the bugs and look at the mockups you’ll see it’s greatly improved. This is a mutually beneficial migration; the personas will be able to leverage AMO features like statistics reporting and collections, and AMO will get a fresh look at reviewing user submitted content and an influx of creative designers.

I snuck in to a personas planning meeting last week and I saw a lot of fun stuff in the pipeline for personas. I’m happy to say migrating them onto AMO will give everyone the server and developer resources to get that new stuff out the door. This will get underway in Q3 of this year.

With the launch of landfill.amo[1] we bypass the entire headache. The site started with a clean database and I uploaded an add-on to show it worked, but otherwise it’s empty. It’s compact, fast, and simple to use. The beauty of the site for volunteers and casual developers is that the database and the filesystem are available in their entirety to download. This means you can check out the code, fill in the configuration, import the landfill database and have the site 90% running.[2]

Perhaps a testament to the obscene number of open bugs for AMO right now, but this also solves a second long standing problem where localizers couldn’t see the entire site. On landfill, anyone can be an administrator, an editor, or any other permission level they’d like; and they’ll be able to see the entire site.

If you’ve been overwhelmed or frustrated trying to set up AMO in the past, now is a good time to give it another shot. The landfill should just get better with age and use – if a few people register and add some data the available database dumps will get richer.

If there is a part of the site that isn’t working and you need it to be, let me know. Keep in mind this is only the new Python code, so the few parts that are still on PHP (like the admin panel) won’t be available until they are ported. Code is updated near-instantly on commit, localization changes are updated every 5 minutes.

[1] Forgive the fake certificate. This is a sandbox for developers, y’all know what you’re doing.

[2] Honestly, 90% is really all you need. We do a lot of stuff for scalability, statistics, etc. and unless you’re actually working on that part of the site, you don’t need those elements running. Of course, you’re more than welcome to turn them on, I’m just trying to make it easy.

In January of 2010 we started migrating addons.mozilla.org from CakePHP to Django. It was a controversial decision. Developers were ambivalent to excited, managers were opposed to neutral – a split anyone would expect. When I first talked about it I expected to be able to turn off PHP by the end of the year. It didn’t turn out quite like that.

Fifteen months later we’re still transitioning and it’s still stressful. The toughest part about a major migration like this is that there is only one team that is doing the migration, continuing to add the new features we need, and all the while maintaining the old site. That’s a stressful environment for developers since the interactions between the languages can be complicated, it’s stressful for managers because features take longer to complete, and it’s stressful for users (and QA for that matter) because issues will arise which are hard to reproduce and complicated to explain.

In the midst of all the work of migration, the rest of the company is still working: the security team is announcing bounties on our site which means we need to be vigilant about fixing issues, project management continues to come up with features to be added, the site perseveres in its never-ending quest for a new look and feel, and Firefox 4 is using AMO like never before meaning approaching 10,000 hits per second is a regular day. All of that is specific to the add-ons site, but consider your own company if you’re thinking of going down the same road – what is coming up for your site that will throw a wrench in the works?

The meat and potatoes of it really comes down to: Given the hindsight of today, would the migration be a good idea? There isn’t a right answer for every site, but for AMO we did the right thing[1]. As of today the majority of pages that matter are on Python – there are some admin tools, and some cron jobs, and the occasional semi-obsolete public page that is on PHP, but for the most part, we’re looking really good (less hand waving, more real data). My new (overly optimistic?) plan is to have PHP off by the end of this year. We’ll see.

To give you an idea of man-hours, we’ve had anywhere from 3 to 6 superhero developers working on the site over the past 15 months, and it’s looking like the whole thing will take around 24 months. That’s a big chunk of time for a site that needs to grow and evolve as quickly as popular sites do these days.

So, overall, I think the lesson is: any reasonably sized site is going to have rabbit holes in it. At first glance AMO might look like it’s got a dozen “main” pages, with a couple dozen more supporting pages (and throw in a few more for the admin CRUD). Have a look at that spreadsheet I linked above and you’ll see that’s not even remotely the case. The spreadsheet even ignores sub-pages in a few places and doesn’t include any new features added in the past year. If you’re considering a migration, think it through well. Make a spreadsheet of every URL, identify the complicated areas, and make sure everyone is clear on the timeline and what it means for new features. People will absolutely try to scope creep your migration – make it clear if a section of the site is migrating as-is or can be migrated and redesigned at the same time. Redesigns add complexity for the developers but can earn you some good will with the users and managers and if you’re in this boat you can use all the good will you can get.

May you have the best of luck with your decisions.

[1] I’ll write another post about pros/cons of the actual frameworks and platforms. Let’s just assume we’re happy with the technical side of the switch for now.

With the launch of the new Add-on Builder (still an alpha product, but usable) I decided to make my first Jetpack to fix this.

In only about 20 lines of code I was able to look for any 4xx or 5xx errors in the HTTP traffic and show a brief notification to the user about what went wrong and where. The builder was a great development experience (lack of documentation aside) and was a breeze to do something relatively complex.

If you’ve wanted a similar add-on, feel free to use Status Watch. It’s a Jetpack so you’ll need Firefox 4, but on the bright side, you won’t even need to restart the browser. Just click and go.

I’ve had some requests to support a whitelist of sites so you don’t get notifications all over the web and I’ll add that when I get time. It’s surprising how many sites have 403s and 404s though.

Update 2011-02-17: Version 2 is on AMO now which supports a whitelist. See the AMO page for details.

My friend sent me a link to conque this morning which I’m so taken with that it’s prompted me to drop coding and write a blog post.

Conque lets me do exactly what I’ve wanted, create shells within Vim. In insert mode you interact with the shell as expected. In normal mode you can scroll back through your history, yank text into buffers, paste and manipulate text. Best thing ever.

Two files open on the left, three shells on the right; iPython, MySQL, and Django’s web server – all with syntax highlighting. ]]> http://micropipes.com/blog/2010/08/13/cli-split-windows-in-vim/feed/ 3 http://micropipes.com/blog/2010/05/03/addons-mozilla-org-%e2%99%a5s-unit-tests-again/ http://micropipes.com/blog/2010/05/03/addons-mozilla-org-%e2%99%a5s-unit-tests-again/#comments Mon, 03 May 2010 17:34:44 +0000 Wil Clouser http://micropipes.com/blog/?p=142 AMO has had an on-again off-again relationship with unit tests. A little over a year ago we had a thousand unit tests that sort of, mostly, ran. The problem is, PHP unit testing just isn’t as good as it should be. CakePHP relies on SimpleTest, one of the main PHP test suites. It worked relatively well for a small number of tests, but as our suite grew, so did our troubles.

Our main issue was hitting a memory limit or the max execution time. We hit the limits often for a variety of reasons, some legitimate bugs, and some because we tried to hack around things to make the tests run. If we change the limits we affect the tests because they are running within the same environment. There wasn’t really a concept of fixtures then, although it looks like CakePHP has stepped up there. The simple test web runner was hard to use and the mock objects were sometimes a little too mocked and missing some attributes.

All in all it was a heroic effort to get that many tests, but we didn’t maintain it because they were so slow to write and difficult to run. Testing can be a pain to write, sure, but it shouldn’t be a burden like that. Enter Django’s testing suite (built on top of Python’s unittest). It has most of our complaints handled out of the box. It’s very well documented, considers a lot of aspects of testing, supports fixtures, a built-in client, etc. It’s a well thought out framework to build tests on.

We’re being more vigilant about requiring tests this time around, but they also aren’t as frustrating to write. When you write them they actually work and they stay working. Most of what you want is built in already. For example, I wrote the password reset form we needed on AMO in Django. With CakePHP and SimpleTest I’d have no idea how to test that the email was actually working. It’s apparently possible with a SimpleTest add-on and enough code that I have to scroll in my browser. With Django’s test suite the actual code was 5 lines, 3 of which were assertions:

    def test_request_success(self):
        self.client.post('/en-US/firefox/users/pwreset',
                        {'email': self.user.email})

        eq_(len(mail.outbox), 1)
        assert mail.outbox[0].subject.find('Password reset') == 0
        assert mail.outbox[0].body.find('pwreset/%s' % self.uidb36) > 0

With the power of the new test suite we’re once again writing and maintaining our unit tests – currently at around 390 tests and increasing steadily. Plenty of people have written about why unit tests are important so I won’t belabor the point, but I will mention that it’s a great feeling to be able to commit something and be confident it hasn’t affected other parts of the site. It’s almost as good of a feeling when you write your code and a completely different test fails pointing out a case that you didn’t even consider but one that would soak up developer time trying to debug down the road.

Building on a foundation that takes testing seriously is great.