We hit some limitations with the NetScaler that we weren’t happy with. Cost aside, it ignored some pretty standard stuff like the HTTP Vary Header. After working around that for years we switched to the horizontally scalable Zeus Traffic Manager (at that time, referred to as ZXTM). We’ve been pleased with our choice and six months ago I wrote a similar PHP library that allows you to connect to Zeus’s API. Time and priorities being what they are, we never implemented it in production.

Finally, the real point to this post, last night I wrote a python library that will expire content from Zeus. We’ll roll this into our migration and waiting on content to expire from Zeus should be a thing of the past.

As always, if you can use the libraries, feel free. They all have READMEs with examples.

Migrating from CakePHP to Django

This is a big undertaking and we’ve been discussing it for quite a while. We’re currently the highest trafficked site on the internet using CakePHP and along with that we’ve run into a lot of frustrating issues. CakePHP has serviced AMO well for several years, so it’s not my intention to bad mouth it here, but I do want to give a fair summary of why we’re moving on. Please also note that AMO is still running on CakePHP 1.1 which is, I think, a year out of date? Three substantial issues:

• Useful Database Abstraction Layer: CakePHP has a concept of database abstraction, but we didn’t find it powerful enough. When it did work it would return enormous nested arrays of data causing massive CPU and memory usage (out of memory errors plague us on AMO). When it didn’t work, we’d end up doing queries directly which kind of defeats the purpose. We couldn’t use prepared statements so we’d have to escape variables ourselves. There was no effective caching built-in and since we just had huge arrays as a response there was no effective way to invalidate the cache we were using (see: Caching is easy; Expiration is hard). The DB layer should return objects that are easy to cache and easy to invalidate. The built-in Django database classes (combined with memcache) should work fine for us here.
• Effective unit tests: I’ve beat the drum about our unit tests before but the simple matter is that it’s really difficult to do them right with the tools we are using. Our test data is already very limited, but if we try to run all our tests right now they’ll run out of memory (and take forever). The CakePHP method of mocking controllers and models was inadequate for what we needed and difficult to deal with. We want our unit tests to run quickly, from the command line, and be independent from each other so there aren’t intermittent problems to waste our time with. We’ll be using Django’s built-in testing framework.
• Better debugging: Debugging in CakePHP amounts to defining a DEBUG level and seeing what is printed on the screen (usually the giant arrays). We supplemented this with Xdebug where we needed it, but that’s still not enough. A framework should have excellent logging and on-the-fly debugging that displays a full traceback (often something will fail deep within CakePHP and we’ll get the file/line where PHP gave up, but not the line in our code that started the problem), the values of variables, the page headers, server settings, SQL that was run, what views and elements are in use, etc. We’re planning on using a combination of pdb, IPython, and the django-debug-toolbar to make all of this easily accessible while developing.

Those are the major issues we’re having right now, but if you want to dig into the comparison some more check out our discussion wiki pages, but realize the majority of discussion happened in person.

Moving away from SVN

We moved AMO into SVN in 2006 and it’s treated us relatively well. Somewhere along the line, we decided to tag our production versions at a revision of trunk instead of keeping a separate tag and merging changes into it. It’s worked for us but it’s a hard cutoff on code changes, which means that while we’re in a code freeze no one can check anything in to trunk. As we begin to branch for larger projects this will become more of a hassle, so I’m planning on going back to a system where a production tag is created and changes are merged into it as they are ready to go live.

Most of the development team has been using git-svn for several months and, aside from the commands being far more verbose, we haven’t had many complaints. We’ve discovered Git is a much more powerful development tool and we expect to use it directly starting some time next year. As of now, we expect to maintain the /locales/ directory in SVN so this change doesn’t affect localizers but we’ll keep people notified if there are any changes to that process.

Continuous Integration

I mentioned excellent testing being one of the reasons we’re moving to Django. Along with that testing is the opportunity for continuous integration. We plan on using Hudson as the framework for our continuous integration. With excellent test coverage and quick feedback from Hudson this should drastically lower our regressions and boost our confidence when we deploy. Speaking of which…

Faster Deployment

For most of 2009 we’ve pushed on 3 week cycles. 2 weeks of development, 1 week of QA and L10n. Delays and regressions being what they are, I think we averaged a little better than a push a month. This is a fairly rapid cycle for a lot of development shops, but I feel like it’s holding us back. We’ve heard a lot of success stories about shorter cycles and I’d like to aim for deployment (optionally, of course) of a few times per week. By shortening the development cycle we reduce the stress of:

• the developers: Everyone likes to see what they’ve done go out quicker and it means less conflicts with others when the patches are smaller.
• the QA team: Right now we dump 2 weeks of work on them and say we need it done right away. With smaller cycles they can verify small changes as they go and not be overwhelmed.
• the infrastructure team: Smaller changes means less to go wrong and with a continuous integration server and some automation they can have minimal involvement with the whole process.
• the localizers: Every time we release we dump a bunch of changes on these fantastic people and tell them we need them back in a week. Most of the time they plow forward and get them done on time. If they don’t though, they are stuck with waiting for the next 3 week cycle. If we push often, it’s not a big deal.
• the product managers: These guys come up with crazy ideas for us to implement and then they stare at graphs and numbers to see if it worked. With shorter cycles they can get faster feedback about what works and what doesn’t.
• the users: Faster release cycles means bugs that are fixed in the repository are fixed on the live site sooner. ’nuff said.

Process Data Offline

Much of AMO relies on cron jobs to get things done. All the statistics, add-on download numbers, how popular an add-on is, all the star rating calculations, any cleanup or maintenance tasks – these are all run via cron and they are so intensive that the database has trouble keeping up. We’re planning on utilizing Gearman to farm all this work out to other machines in incremental pieces instead of single huge queries. Any heavy calculating that can be done offline will be moved to these external processors which should help improve the speed of the site and make all our statistics more reliable (as currently the cron jobs have a tendency to fail before they are complete).

Improve the Documentation

Documentation is a noble goal of many developers but it rarely gets enough attention. We evaluated our current documentation and found it is woefully out of date. By being on a wiki that is rarely used it doesn’t get updated except when someone tries to use it and sees it’s not right. We’re hoping to change that by moving the developer documentation into the code repository itself. We’ll be able to integrate with generated API docs, style the docs however we want, and check in changes right along with our code patches. When someone checks out a copy of AMO, they’ll get all the documentation right along with it. We’ll use Sphinx to build the docs.

The outline above details several large, high-level changes but there are a lot of other plans for smaller improvements as well. This post got a lot longer than I was expecting, but I’m really excited about the direction AMO is headed for 2010. As these changes are implemented the site will become more responsive and reliable, and we’ll be able to adapt to the needs of Mozilla’s users even faster. As always, feedback and discussion are welcome and stay tuned for further back end improvements.

To get an object out of the cache in the previous version the method signature was:

getcacheobject(string $url, string $host, unsignedInt $port)

$url in this case is what is commonly referred to as the “path” in a standard URL; e.g. /en-US/index.html. That’s simple enough to use, let’s see what the new version has:

getcacheobject(string $url, unsignedLong $locator, string $host, unsignedInt $port [...]

$locator is an internal id for a cache object which is defintitely something I should be able to use to expire content, but what is it doing in the middle of my url+host pair? The error messages are happy to let me know that I can’t pass both $url and $locator to the same call and if I pass $url I must pass $host. That means I will always have be passing a null value in either the first or second parameter.

And if you’re thinking it’s for consistency with other methods, think again. To flush an object we’ve got:

flushcacheobject(unsignedLong $locator, string $url, string $host, unsignedInt $port [...]

Moral of the story: planning and consistency makes happy programmers; particularly with an API.

This site accepts a URL which it loads over a network connection with the speed of your choice. After loading the site it offers you a waterfall graph of page elements, a screenshot so you can see if it loaded correctly, and automatic comparisons to previous page loads to see how caching affects your site.

The code behind the page is based on the open source AOL Pagetest which means it can be modified as needed. I think I heard Ryan say something about automated testing over time with historical graphs and email alerts?

I noticed their first best practice for memcached, “Don’t think row-level (database) caching, think complex objects, ” is in opposition with what we’re doing on AMO. After stuff like bug 425315 I completely agree with them.

The problem we’re running into in the bug linked above is knowing when to expire a cache. Consider when an add-on author updates the summary of their add-on. We know we’ll have to flush the queries on that page out of memcached, and that’s easy enough, but what about all the other places the summary is used? Search results pages, add-on detail pages, recommended lists, the API, etc. Now we’ve got to figure out the queries used on those pages and expire them too. Suddenly I’m wishing we were caching objects in memcached instead of queries.

I looked in to other ways to use memcached and they all have their pros and cons. Caching entire pages means we’d have to store different versions for a person that is logged in vs. logged out and also what permissions they had (pages have different options for localizers, admins, developers, etc.). Caching objects is attractive, but the way CakePHP does queries makes this a non-option (namely, it’s not asking objects for values, it does joins directly on the db). Directly caching queries seems like the best fit because we can affect just the parts of the pages we want and it will work with CakePHP’s current system…just as soon as we figure out how to relate updating a row to all of it’s associated queries.

I attached an idea to the bug but regardless of the process we use, figuring out how to implement a full time cache that we can expire on the fly is going to be an important step in keeping the AMO site usable as our traffic increases.

Version 1 of AMO wasn’t concerned with caching. It was straight PHP talking directly to a single MySQL box. Short, easy, and not very scalable.

Version 2 of AMO progressed through several caching systems. The site used the Smarty template engine so our first step was to turn on the built in Smarty cache. That didn’t give us the performance we needed, so Mike Morgan started caching page output in PEAR’s Cache_Lite. I don’t remember the specifics of this implementation since it was so short lived (less than a month), but the CVS log, mentions problems with “scalability in a clustered environment.” Our next step was to store the same page output in memcached instead of Cache_Lite which brought pretty satisfying results. Thus began our abuse of memcached.

In addition to memcached and expanding the number of web servers it ran on, version 2 also boasted two other significant performance improvements. The first was the ability to talk to a slave database for read-only queries which, when combined with a load balancer, let us scale database servers horizontally. The second was installing a NetScaler in front of addons.mozilla.org giving us the benefits of a reverse proxy cache and SSL offloading. These changes bought us precious time when hoards of Firefox 1.5 users were clamoring for add-ons. In fact, I’d say we were in pretty good shape at that point.

Fast forward to Version 3 (the current version). We’ve expanded the memcache servers from one to two and instead of page output we’re storing database queries and their results. We’re still using a single master database but are using two slaves now for read only queries. There are several NetScalers around the world caching pages locally[1] for closer regions. We’ve survived quite a while on this system but we’re starting to push the envelope again and we’re going to need to make some changes to be able to scale for Firefox 3 and still provide a good user experience. I’ll write more about our plans as they develop.

[1] Users who are logged in to AMO don’t get the local caches – their connection is always to San Jose, CA.

Today’s subject is anonymous sessions. Frameworks (and developers) love them because they are simple and convenient, but it comes at a cost. Keeping track of sessions for every visitor on a high traffic site is far too expensive to be practical. Developers should know how to work around this, but their frameworks need to support them.

The first framework on my mind is Drupal. I filed an issue last year that Drupal should support disabling anonymous sessions. It’s still unassigned so I’m guessing it’s not a high priority, but it was one of the main things that made me choose not to use Drupal on mozilla.com. I wrote some ideas on how to handle it and got some responses from people suffering the same fate. No word on any progress though.

The second framework, CakePHP, has an AUTO_SESSION variable that, just like $cacheQueries, is far to easy to misplace faith in.

By setting AUTO_SESSION to false, you can’t read or write to the session. Working as advertised? Not so much. If you take a closer look at what’s actually happening you’ll see that the session is still getting started, it’s just that CakePHP is blocking your access to it. Even with AUTO_SESSION off, a cookie with a unique ID is set, and a row is still inserted into the sessions table. That last part almost brought down AMO last night. I wrote a patch that disables anonymous sessions for real, but anyone that has talked to me about patching core code knows I don’t like to do it.

When you’re writing code, framework or not, don’t forget about scalability.