With the launch of landfill.amo[1] we bypass the entire headache. The site started with a clean database and I uploaded an add-on to show it worked, but otherwise it’s empty. It’s compact, fast, and simple to use. The beauty of the site for volunteers and casual developers is that the database and the filesystem are available in their entirety to download. This means you can check out the code, fill in the configuration, import the landfill database and have the site 90% running.[2]

Perhaps a testament to the obscene number of open bugs for AMO right now, but this also solves a second long standing problem where localizers couldn’t see the entire site. On landfill, anyone can be an administrator, an editor, or any other permission level they’d like; and they’ll be able to see the entire site.

If you’ve been overwhelmed or frustrated trying to set up AMO in the past, now is a good time to give it another shot. The landfill should just get better with age and use – if a few people register and add some data the available database dumps will get richer.

If there is a part of the site that isn’t working and you need it to be, let me know. Keep in mind this is only the new Python code, so the few parts that are still on PHP (like the admin panel) won’t be available until they are ported. Code is updated near-instantly on commit, localization changes are updated every 5 minutes.

[1] Forgive the fake certificate. This is a sandbox for developers, y’all know what you’re doing.

[2] Honestly, 90% is really all you need. We do a lot of stuff for scalability, statistics, etc. and unless you’re actually working on that part of the site, you don’t need those elements running. Of course, you’re more than welcome to turn them on, I’m just trying to make it easy.

In January of 2010 we started migrating addons.mozilla.org from CakePHP to Django. It was a controversial decision. Developers were ambivalent to excited, managers were opposed to neutral – a split anyone would expect. When I first talked about it I expected to be able to turn off PHP by the end of the year. It didn’t turn out quite like that.

Fifteen months later we’re still transitioning and it’s still stressful. The toughest part about a major migration like this is that there is only one team that is doing the migration, continuing to add the new features we need, and all the while maintaining the old site. That’s a stressful environment for developers since the interactions between the languages can be complicated, it’s stressful for managers because features take longer to complete, and it’s stressful for users (and QA for that matter) because issues will arise which are hard to reproduce and complicated to explain.

In the midst of all the work of migration, the rest of the company is still working: the security team is announcing bounties on our site which means we need to be vigilant about fixing issues, project management continues to come up with features to be added, the site perseveres in its never-ending quest for a new look and feel, and Firefox 4 is using AMO like never before meaning approaching 10,000 hits per second is a regular day. All of that is specific to the add-ons site, but consider your own company if you’re thinking of going down the same road – what is coming up for your site that will throw a wrench in the works?

The meat and potatoes of it really comes down to: Given the hindsight of today, would the migration be a good idea? There isn’t a right answer for every site, but for AMO we did the right thing[1]. As of today the majority of pages that matter are on Python – there are some admin tools, and some cron jobs, and the occasional semi-obsolete public page that is on PHP, but for the most part, we’re looking really good (less hand waving, more real data). My new (overly optimistic?) plan is to have PHP off by the end of this year. We’ll see.

To give you an idea of man-hours, we’ve had anywhere from 3 to 6 superhero developers working on the site over the past 15 months, and it’s looking like the whole thing will take around 24 months. That’s a big chunk of time for a site that needs to grow and evolve as quickly as popular sites do these days.

So, overall, I think the lesson is: any reasonably sized site is going to have rabbit holes in it. At first glance AMO might look like it’s got a dozen “main” pages, with a couple dozen more supporting pages (and throw in a few more for the admin CRUD). Have a look at that spreadsheet I linked above and you’ll see that’s not even remotely the case. The spreadsheet even ignores sub-pages in a few places and doesn’t include any new features added in the past year. If you’re considering a migration, think it through well. Make a spreadsheet of every URL, identify the complicated areas, and make sure everyone is clear on the timeline and what it means for new features. People will absolutely try to scope creep your migration – make it clear if a section of the site is migrating as-is or can be migrated and redesigned at the same time. Redesigns add complexity for the developers but can earn you some good will with the users and managers and if you’re in this boat you can use all the good will you can get.

May you have the best of luck with your decisions.

[1] I’ll write another post about pros/cons of the actual frameworks and platforms. Let’s just assume we’re happy with the technical side of the switch for now.

We hit some limitations with the NetScaler that we weren’t happy with. Cost aside, it ignored some pretty standard stuff like the HTTP Vary Header. After working around that for years we switched to the horizontally scalable Zeus Traffic Manager (at that time, referred to as ZXTM). We’ve been pleased with our choice and six months ago I wrote a similar PHP library that allows you to connect to Zeus’s API. Time and priorities being what they are, we never implemented it in production.

Finally, the real point to this post, last night I wrote a python library that will expire content from Zeus. We’ll roll this into our migration and waiting on content to expire from Zeus should be a thing of the past.

As always, if you can use the libraries, feel free. They all have READMEs with examples.

First out of the chute was the .po files. It turns out the actual formatting is different between the two languages. PHP uses %1$s for its substitutions, but python uses either named variables like (num)s or integers like {0}. For the record, they both support %s when you don’t need to order the substitutions.
PHP example:
I have %2$s apples and %1$s oranges
Python example:
I have {1} apples and {0} oranges

Since I’ve worked with the Translate Toolkit before, I decided to write a script to convert between the two formats. If you find yourself in the same unfortunate boat as me, behold
phppo2pypo and pypo2phppo to convert between the two types.

Crisis averted, right? Oh, that’s just scratching the surface. Remember how happy I was that PHP finally started supporting msgctxt? Well, Python has had a patch for it since 2008 but no one has bothered to land it. I wrote a new ugettext() and ungettext() that recognizes context in the .po files. To use simply do: from l10n import ugettext as _ at the top of your file.

Along with adding msgctxt support, those two functions also collapse consecutive white space. We’re using Jinja2 with Babel and the i18n extension as our template engine. Jinja2 has a concept of stripping white space from the beginning or end of a string but does nothing about the middle. A paragraph of text in a Jinja2 template would look like:

{% trans -%}Mozilla is providing links to these applications
as a courtesy, and makes no representations regarding the
applications or any information related thereto. Any questions,
complaints or claims regarding the applications must be
directed to the appropriate software vendor.
{%- endtrans %}


That’s a decent looking template, right? Yeah, well, when Babel extracts that, it includes all the line breaks too, giving you something like this. The localizers would revolt if I sent them that, so I added in auto white-space collapsing. Getting Babel to use the new functions means a new extraction script.

At this point, we’re extracting strings from our new code and we can convert between Python and PHP files. All we need now is a Frankenstein mix of xgettext functions to act as glue. Meet the amalgamate script that uses the pypo2php scripts, concatenates the .pot files, and merge updates each locales .po file. After that it’s quick tweaks to the build scripts to create z-messages.po files and we’re done.

So, all that said, the new process for L10n, while we’re in this transitional phase, is:

1. From the PHP code, run locale/extract-po-remora.sh. That pulls everything from all the PHP files, creates locale/r-keys.pot, updates the messages.po file for each locale, and compiles them. Life used to be so simple.
2. From the python code, make sure you’re up to date, then run ./manage.py extract. That will pull everything from the python code and templates and create locale/z-keys.pot.
3. Run ./manage.py amalgamate. That will merge the z-keys.pot into the PHP messages.po files.
4. Localizers can make their changes as usual, and commit back to messages.po.
5. From PHP, locale/copy-to-zamboni.py locale will create z-messages.po files in the Python format. We could skip right to .mo files, but in case something goes wrong I want to see the .po files.
6. Then, like today, locale/compile-mo.sh locale will compile all the .po files.

After all those steps are done, we’ve got duplicate .mo files, aside from formatting, and each application can look at its own .mo to get the strings it needs. All this code is just a big band-aid and there are plenty of things that are more fun than juggling L10n between two applications across two RCSs. But we knew what we were getting in to. I’ll post something more positive later to help justify it.

Migrating from CakePHP to Django

This is a big undertaking and we’ve been discussing it for quite a while. We’re currently the highest trafficked site on the internet using CakePHP and along with that we’ve run into a lot of frustrating issues. CakePHP has serviced AMO well for several years, so it’s not my intention to bad mouth it here, but I do want to give a fair summary of why we’re moving on. Please also note that AMO is still running on CakePHP 1.1 which is, I think, a year out of date? Three substantial issues:

• Useful Database Abstraction Layer: CakePHP has a concept of database abstraction, but we didn’t find it powerful enough. When it did work it would return enormous nested arrays of data causing massive CPU and memory usage (out of memory errors plague us on AMO). When it didn’t work, we’d end up doing queries directly which kind of defeats the purpose. We couldn’t use prepared statements so we’d have to escape variables ourselves. There was no effective caching built-in and since we just had huge arrays as a response there was no effective way to invalidate the cache we were using (see: Caching is easy; Expiration is hard). The DB layer should return objects that are easy to cache and easy to invalidate. The built-in Django database classes (combined with memcache) should work fine for us here.
• Effective unit tests: I’ve beat the drum about our unit tests before but the simple matter is that it’s really difficult to do them right with the tools we are using. Our test data is already very limited, but if we try to run all our tests right now they’ll run out of memory (and take forever). The CakePHP method of mocking controllers and models was inadequate for what we needed and difficult to deal with. We want our unit tests to run quickly, from the command line, and be independent from each other so there aren’t intermittent problems to waste our time with. We’ll be using Django’s built-in testing framework.
• Better debugging: Debugging in CakePHP amounts to defining a DEBUG level and seeing what is printed on the screen (usually the giant arrays). We supplemented this with Xdebug where we needed it, but that’s still not enough. A framework should have excellent logging and on-the-fly debugging that displays a full traceback (often something will fail deep within CakePHP and we’ll get the file/line where PHP gave up, but not the line in our code that started the problem), the values of variables, the page headers, server settings, SQL that was run, what views and elements are in use, etc. We’re planning on using a combination of pdb, IPython, and the django-debug-toolbar to make all of this easily accessible while developing.

Those are the major issues we’re having right now, but if you want to dig into the comparison some more check out our discussion wiki pages, but realize the majority of discussion happened in person.

Moving away from SVN

We moved AMO into SVN in 2006 and it’s treated us relatively well. Somewhere along the line, we decided to tag our production versions at a revision of trunk instead of keeping a separate tag and merging changes into it. It’s worked for us but it’s a hard cutoff on code changes, which means that while we’re in a code freeze no one can check anything in to trunk. As we begin to branch for larger projects this will become more of a hassle, so I’m planning on going back to a system where a production tag is created and changes are merged into it as they are ready to go live.

Most of the development team has been using git-svn for several months and, aside from the commands being far more verbose, we haven’t had many complaints. We’ve discovered Git is a much more powerful development tool and we expect to use it directly starting some time next year. As of now, we expect to maintain the /locales/ directory in SVN so this change doesn’t affect localizers but we’ll keep people notified if there are any changes to that process.

Continuous Integration

I mentioned excellent testing being one of the reasons we’re moving to Django. Along with that testing is the opportunity for continuous integration. We plan on using Hudson as the framework for our continuous integration. With excellent test coverage and quick feedback from Hudson this should drastically lower our regressions and boost our confidence when we deploy. Speaking of which…

Faster Deployment

For most of 2009 we’ve pushed on 3 week cycles. 2 weeks of development, 1 week of QA and L10n. Delays and regressions being what they are, I think we averaged a little better than a push a month. This is a fairly rapid cycle for a lot of development shops, but I feel like it’s holding us back. We’ve heard a lot of success stories about shorter cycles and I’d like to aim for deployment (optionally, of course) of a few times per week. By shortening the development cycle we reduce the stress of:

• the developers: Everyone likes to see what they’ve done go out quicker and it means less conflicts with others when the patches are smaller.
• the QA team: Right now we dump 2 weeks of work on them and say we need it done right away. With smaller cycles they can verify small changes as they go and not be overwhelmed.
• the infrastructure team: Smaller changes means less to go wrong and with a continuous integration server and some automation they can have minimal involvement with the whole process.
• the localizers: Every time we release we dump a bunch of changes on these fantastic people and tell them we need them back in a week. Most of the time they plow forward and get them done on time. If they don’t though, they are stuck with waiting for the next 3 week cycle. If we push often, it’s not a big deal.
• the product managers: These guys come up with crazy ideas for us to implement and then they stare at graphs and numbers to see if it worked. With shorter cycles they can get faster feedback about what works and what doesn’t.
• the users: Faster release cycles means bugs that are fixed in the repository are fixed on the live site sooner. ’nuff said.

Process Data Offline

Much of AMO relies on cron jobs to get things done. All the statistics, add-on download numbers, how popular an add-on is, all the star rating calculations, any cleanup or maintenance tasks – these are all run via cron and they are so intensive that the database has trouble keeping up. We’re planning on utilizing Gearman to farm all this work out to other machines in incremental pieces instead of single huge queries. Any heavy calculating that can be done offline will be moved to these external processors which should help improve the speed of the site and make all our statistics more reliable (as currently the cron jobs have a tendency to fail before they are complete).

Improve the Documentation

Documentation is a noble goal of many developers but it rarely gets enough attention. We evaluated our current documentation and found it is woefully out of date. By being on a wiki that is rarely used it doesn’t get updated except when someone tries to use it and sees it’s not right. We’re hoping to change that by moving the developer documentation into the code repository itself. We’ll be able to integrate with generated API docs, style the docs however we want, and check in changes right along with our code patches. When someone checks out a copy of AMO, they’ll get all the documentation right along with it. We’ll use Sphinx to build the docs.

The outline above details several large, high-level changes but there are a lot of other plans for smaller improvements as well. This post got a lot longer than I was expecting, but I’m really excited about the direction AMO is headed for 2010. As these changes are implemented the site will become more responsive and reliable, and we’ll be able to adapt to the needs of Mozilla’s users even faster. As always, feedback and discussion are welcome and stay tuned for further back end improvements.

I’m not trying to invite attackers by posting this. My goal is to provide a (hopefully) working example from a real world, high-traffic site. I think the people exploiting XSS have a fairly good idea what they are doing and, too often, the people attempting to secure their sites don’t. Since AMO is open source I’m not sharing anything that isn’t available already anyway (side note: please don’t depend on security by obscurity).

Firstly, this chunk of code sits in CakePHP’s bootstrap.php and runs very close to the start of every request:

if (array_key_exists('url',$_GET) &&
    !preg_match('/\/api\//', $_GET['url']) &&
    preg_match('/[^\w\d\/\.\-_!: ]/u',$_GET['url'])) {
    header("HTTP/1.1 400 Bad Request");
    exit;
}

Since a lot of XSS attacks are launched from the URL we implemented this simple white list of characters we’ll allow. If anything outside of that white-list is in the URL we return an invalid request header and die. This isn’t a lot of protection but it does narrow the field of what our application expects and has to deal with (particularly with control characters, high level ASCII, etc.).

The second, and more important section of code is in our app_controller class. We wrote a custom sanitize() function that any string going into one of our views gets run through:

$sanitize_patterns = array(
    'patterns'      => array("/%/u", "/\(/u", "/\)/u", "/\+/u", "/-/u"),
    'replacements'  => array("%", "(", ")", "+", "-")
    );

........

$data = iconv('UTF-8', 'UTF-8//IGNORE', $data);
$data = htmlspecialchars($data, ENT_QUOTES, 'UTF-8');
$data = preg_replace($sanitize_patterns['patterns'], $sanitize_patterns['replacements'], $data);

This code has several important parts and I’ll start with the functions. The first function that modifies the actual data is iconv(). We ask it to convert our data from UTF-8 to UTF-8 which seems unnecessary but the “//IGNORE” part is important – that means it will throw out any characters it can’t represent appropriately. This was added to prevent a proof of concept attack that exploited a C0 ASCII control code character to break the output (discovered on the sla.ckers.org forums).

The next function, htmlspecialchars(), is a pretty well known function and converts special characters to their ASCII equivalents. The second parameter specifically asks it to encode single quotes.

Lastly we use the array of patterns and replacements declared at the beginning to encode a few final symbols, like parenthesis and the percentage sign, into HTML entities.

This system has worked fairly well for a few years now and as issues are discovered we make changes to it. If you’re looking for the latest code please be sure to check our repository. And, as always, if you find any kind of exploit on AMO please let me know!

Version 1 of AMO wasn’t concerned with caching. It was straight PHP talking directly to a single MySQL box. Short, easy, and not very scalable.

Version 2 of AMO progressed through several caching systems. The site used the Smarty template engine so our first step was to turn on the built in Smarty cache. That didn’t give us the performance we needed, so Mike Morgan started caching page output in PEAR‘s Cache_Lite. I don’t remember the specifics of this implementation since it was so short lived (less than a month), but the CVS log, mentions problems with “scalability in a clustered environment.” Our next step was to store the same page output in memcached instead of Cache_Lite which brought pretty satisfying results. Thus began our abuse of memcached.

In addition to memcached and expanding the number of web servers it ran on, version 2 also boasted two other significant performance improvements. The first was the ability to talk to a slave database for read-only queries which, when combined with a load balancer, let us scale database servers horizontally. The second was installing a NetScaler in front of addons.mozilla.org giving us the benefits of a reverse proxy cache and SSL offloading. These changes bought us precious time when hoards of Firefox 1.5 users were clamoring for add-ons. In fact, I’d say we were in pretty good shape at that point.

Fast forward to Version 3 (the current version). We’ve expanded the memcache servers from one to two and instead of page output we’re storing database queries and their results. We’re still using a single master database but are using two slaves now for read only queries. There are several NetScalers around the world caching pages locally[1] for closer regions. We’ve survived quite a while on this system but we’re starting to push the envelope again and we’re going to need to make some changes to be able to scale for Firefox 3 and still provide a good user experience. I’ll write more about our plans as they develop.

[1] Users who are logged in to AMO don’t get the local caches – their connection is always to San Jose, CA.

First, some background:

The uniqueness of a cookie in the browser is determined by all the attributes when it’s set (not just it’s name). That means I can have multiple cookies named AMOv3 as long as another attribute (eg. path) is different when I set the second cookie.

According to RFC 2109:

If multiple cookies satisfy the criteria [...] they are ordered in the Cookie header such that those with more specific Path attributes precede those with less specific.

In PHP, however, cookies are indexed in the $_COOKIE variable by name, which means I can send several cookies with the same name and only the first cookie will show up. ^[1]

So, what’s this have to do with AMO? For some reason, CakePHP hasn’t been consistent in the past regarding cookie paths. Sometimes they were / (which is correct), sometimes they were something else like /en-US/firefox/users/ and sometimes users had multiple cookies with different paths.

From the PHP manual:

Cookies must be deleted with the same parameters as they were set with. If the value argument is an empty string, or FALSE, and all other arguments match a previous call to setcookie, then the cookie with the specified name will be deleted from the remote client.

When a browser sends a cookie back to a server it only sends the name and value. If a cookie was set with a path that I don’t know I have no way to remove that cookie!

To compound the problem, if a cookie with the same name has a more specific path, it shows up in $_COOKIE and there is no indication the other cookie even exists. This means on the front page your session can be fine and after clicking on a deeper URL you’re suddenly requesting a session that has been expired long ago.

We rolled out a change to session handling last week that prevented myself and another developer from logging in because we had a mess of cookies with different paths. I haven’t had a barrage of emails so I don’t think it’s affected many people, but if you have any troubles with logging in please let me know. If you’re in a hurry, clearing your cookies for the addons.mozilla.org domain will fix any problems.

For the record, any new login cookies on AMO will expire when the browser closes which should prevent stale cookies from coming back to haunt us.

[1] If you need to get at all the cookies the browser is sending, you’ll have to dig through $_SERVER['HTTP_COOKIE'] manually.