Our main issue was hitting a memory limit or the max execution time. We hit the limits often for a variety of reasons, some legitimate bugs, and some because we tried to hack around things to make the tests run. If we change the limits we affect the tests because they are running within the same environment. There wasn’t really a concept of fixtures then, although it looks like CakePHP has stepped up there. The simple test web runner was hard to use and the mock objects were sometimes a little too mocked and missing some attributes.

All in all it was a heroic effort to get that many tests, but we didn’t maintain it because they were so slow to write and difficult to run. Testing can be a pain to write, sure, but it shouldn’t be a burden like that. Enter Django’s testing suite (built on top of Python’s unittest). It has most of our complaints handled out of the box. It’s very well documented, considers a lot of aspects of testing, supports fixtures, a built-in client, etc. It’s a well thought out framework to build tests on.

We’re being more vigilant about requiring tests this time around, but they also aren’t as frustrating to write. When you write them they actually work and they stay working. Most of what you want is built in already. For example, I wrote the password reset form we needed on AMO in Django. With CakePHP and SimpleTest I’d have no idea how to test that the email was actually working. It’s apparently possible with a SimpleTest add-on and enough code that I have to scroll in my browser. With Django’s test suite the actual code was 5 lines, 3 of which were assertions:

    def test_request_success(self):
        self.client.post('/en-US/firefox/users/pwreset',
                        {'email': self.user.email})

        eq_(len(mail.outbox), 1)
        assert mail.outbox[0].subject.find('Password reset') == 0
        assert mail.outbox[0].body.find('pwreset/%s' % self.uidb36) > 0

With the power of the new test suite we’re once again writing and maintaining our unit tests – currently at around 390 tests and increasing steadily. Plenty of people have written about why unit tests are important so I won’t belabor the point, but I will mention that it’s a great feeling to be able to commit something and be confident it hasn’t affected other parts of the site. It’s almost as good of a feeling when you write your code and a completely different test fails pointing out a case that you didn’t even consider but one that would soak up developer time trying to debug down the road.

Building on a foundation that takes testing seriously is great.

Migrating from CakePHP to Django

This is a big undertaking and we’ve been discussing it for quite a while. We’re currently the highest trafficked site on the internet using CakePHP and along with that we’ve run into a lot of frustrating issues. CakePHP has serviced AMO well for several years, so it’s not my intention to bad mouth it here, but I do want to give a fair summary of why we’re moving on. Please also note that AMO is still running on CakePHP 1.1 which is, I think, a year out of date? Three substantial issues:

• Useful Database Abstraction Layer: CakePHP has a concept of database abstraction, but we didn’t find it powerful enough. When it did work it would return enormous nested arrays of data causing massive CPU and memory usage (out of memory errors plague us on AMO). When it didn’t work, we’d end up doing queries directly which kind of defeats the purpose. We couldn’t use prepared statements so we’d have to escape variables ourselves. There was no effective caching built-in and since we just had huge arrays as a response there was no effective way to invalidate the cache we were using (see: Caching is easy; Expiration is hard). The DB layer should return objects that are easy to cache and easy to invalidate. The built-in Django database classes (combined with memcache) should work fine for us here.
• Effective unit tests: I’ve beat the drum about our unit tests before but the simple matter is that it’s really difficult to do them right with the tools we are using. Our test data is already very limited, but if we try to run all our tests right now they’ll run out of memory (and take forever). The CakePHP method of mocking controllers and models was inadequate for what we needed and difficult to deal with. We want our unit tests to run quickly, from the command line, and be independent from each other so there aren’t intermittent problems to waste our time with. We’ll be using Django’s built-in testing framework.
• Better debugging: Debugging in CakePHP amounts to defining a DEBUG level and seeing what is printed on the screen (usually the giant arrays). We supplemented this with Xdebug where we needed it, but that’s still not enough. A framework should have excellent logging and on-the-fly debugging that displays a full traceback (often something will fail deep within CakePHP and we’ll get the file/line where PHP gave up, but not the line in our code that started the problem), the values of variables, the page headers, server settings, SQL that was run, what views and elements are in use, etc. We’re planning on using a combination of pdb, IPython, and the django-debug-toolbar to make all of this easily accessible while developing.

Those are the major issues we’re having right now, but if you want to dig into the comparison some more check out our discussion wiki pages, but realize the majority of discussion happened in person.

Moving away from SVN

We moved AMO into SVN in 2006 and it’s treated us relatively well. Somewhere along the line, we decided to tag our production versions at a revision of trunk instead of keeping a separate tag and merging changes into it. It’s worked for us but it’s a hard cutoff on code changes, which means that while we’re in a code freeze no one can check anything in to trunk. As we begin to branch for larger projects this will become more of a hassle, so I’m planning on going back to a system where a production tag is created and changes are merged into it as they are ready to go live.

Most of the development team has been using git-svn for several months and, aside from the commands being far more verbose, we haven’t had many complaints. We’ve discovered Git is a much more powerful development tool and we expect to use it directly starting some time next year. As of now, we expect to maintain the /locales/ directory in SVN so this change doesn’t affect localizers but we’ll keep people notified if there are any changes to that process.

Continuous Integration

I mentioned excellent testing being one of the reasons we’re moving to Django. Along with that testing is the opportunity for continuous integration. We plan on using Hudson as the framework for our continuous integration. With excellent test coverage and quick feedback from Hudson this should drastically lower our regressions and boost our confidence when we deploy. Speaking of which…

Faster Deployment

For most of 2009 we’ve pushed on 3 week cycles. 2 weeks of development, 1 week of QA and L10n. Delays and regressions being what they are, I think we averaged a little better than a push a month. This is a fairly rapid cycle for a lot of development shops, but I feel like it’s holding us back. We’ve heard a lot of success stories about shorter cycles and I’d like to aim for deployment (optionally, of course) of a few times per week. By shortening the development cycle we reduce the stress of:

• the developers: Everyone likes to see what they’ve done go out quicker and it means less conflicts with others when the patches are smaller.
• the QA team: Right now we dump 2 weeks of work on them and say we need it done right away. With smaller cycles they can verify small changes as they go and not be overwhelmed.
• the infrastructure team: Smaller changes means less to go wrong and with a continuous integration server and some automation they can have minimal involvement with the whole process.
• the localizers: Every time we release we dump a bunch of changes on these fantastic people and tell them we need them back in a week. Most of the time they plow forward and get them done on time. If they don’t though, they are stuck with waiting for the next 3 week cycle. If we push often, it’s not a big deal.
• the product managers: These guys come up with crazy ideas for us to implement and then they stare at graphs and numbers to see if it worked. With shorter cycles they can get faster feedback about what works and what doesn’t.
• the users: Faster release cycles means bugs that are fixed in the repository are fixed on the live site sooner. ’nuff said.

Process Data Offline

Much of AMO relies on cron jobs to get things done. All the statistics, add-on download numbers, how popular an add-on is, all the star rating calculations, any cleanup or maintenance tasks – these are all run via cron and they are so intensive that the database has trouble keeping up. We’re planning on utilizing Gearman to farm all this work out to other machines in incremental pieces instead of single huge queries. Any heavy calculating that can be done offline will be moved to these external processors which should help improve the speed of the site and make all our statistics more reliable (as currently the cron jobs have a tendency to fail before they are complete).

Improve the Documentation

Documentation is a noble goal of many developers but it rarely gets enough attention. We evaluated our current documentation and found it is woefully out of date. By being on a wiki that is rarely used it doesn’t get updated except when someone tries to use it and sees it’s not right. We’re hoping to change that by moving the developer documentation into the code repository itself. We’ll be able to integrate with generated API docs, style the docs however we want, and check in changes right along with our code patches. When someone checks out a copy of AMO, they’ll get all the documentation right along with it. We’ll use Sphinx to build the docs.

The outline above details several large, high-level changes but there are a lot of other plans for smaller improvements as well. This post got a lot longer than I was expecting, but I’m really excited about the direction AMO is headed for 2010. As these changes are implemented the site will become more responsive and reliable, and we’ll be able to adapt to the needs of Mozilla’s users even faster. As always, feedback and discussion are welcome and stay tuned for further back end improvements.

A tag library exists for CakePHP but it, and many others, are too simplistic for what we want.

We’ve written our tagging goals into a plan but have some technical details we still need to figure out. While reviewing what we have a couple questions arose that we thought people would have opinions on.

1) What should the range of allowed characters be? Our first instinct was simplicity, something like /[A-Za-z0-9-]/ (that is, all English letters and numbers and a dash). This is easy to handle on our end but leaves out everyone that doesn’t want to add tags using the English alphabet. There is some debate how useful it would be to allow other Unicode characters, particularly when you think about #2 below.

2) Tags are most useful when they are normalized. By allowing Unicode characters we run the risk of diluting our tag cloud. For example, resume and résumé are close enough that for our purposes they are equivalent. If we allow Unicode we’ll have to deal with converting characters like é to e and vice versa for searches. At that point we’ll need a list of “equivalent” characters – not impossible but it will slow things down (both development and speed of a search). The second question is: Assuming you think we should allow Unicode characters, what characters are equivalents? Here is a quick idea from php.net’s strtr() documentation:

$a = 'ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûýýþÿŔŕ';
$b = 'aaaaaaaceeeeiiiidnoooooouuuuybsaaaaaaaceeeeiiiidnoooooouuuyybyRr';

Some other aspects of our current plan are:

• Tags are not localizable in the same way as other strings on the site (like categories). There isn’t anything stopping someone from using “WebDev” as a tag or creating a new tag with “WebDev” translated in their language. However, there won’t be any relationship between the two translated tags.
• Tags are separated by spaces. Spaces within tags are allowed with quotes.
• Spaces will be preserved when displaying a tag on the add-on’s page, however, they will be removed for displaying the tag in a URL and for doing logical operations on the back end like searching. This means searching for “Portland OR” will actually be collapsed to “PortlandOR” and will match either “Portland OR” or “PortlandOR” tags. This is consistent with flickr.
• If unicode is allowed we’ll preserve characters as they are entered even if we are actually searching on their “equivalents.”

I’m not trying to invite attackers by posting this. My goal is to provide a (hopefully) working example from a real world, high-traffic site. I think the people exploiting XSS have a fairly good idea what they are doing and, too often, the people attempting to secure their sites don’t. Since AMO is open source I’m not sharing anything that isn’t available already anyway (side note: please don’t depend on security by obscurity).

Firstly, this chunk of code sits in CakePHP’s bootstrap.php and runs very close to the start of every request:

if (array_key_exists('url',$_GET) &&
    !preg_match('/\/api\//', $_GET['url']) &&
    preg_match('/[^\w\d\/\.\-_!: ]/u',$_GET['url'])) {
    header("HTTP/1.1 400 Bad Request");
    exit;
}

Since a lot of XSS attacks are launched from the URL we implemented this simple white list of characters we’ll allow. If anything outside of that white-list is in the URL we return an invalid request header and die. This isn’t a lot of protection but it does narrow the field of what our application expects and has to deal with (particularly with control characters, high level ASCII, etc.).

The second, and more important section of code is in our app_controller class. We wrote a custom sanitize() function that any string going into one of our views gets run through:

$sanitize_patterns = array(
    'patterns'      => array("/%/u", "/\(/u", "/\)/u", "/\+/u", "/-/u"),
    'replacements'  => array("%", "(", ")", "+", "-")
    );

........

$data = iconv('UTF-8', 'UTF-8//IGNORE', $data);
$data = htmlspecialchars($data, ENT_QUOTES, 'UTF-8');
$data = preg_replace($sanitize_patterns['patterns'], $sanitize_patterns['replacements'], $data);

This code has several important parts and I’ll start with the functions. The first function that modifies the actual data is iconv(). We ask it to convert our data from UTF-8 to UTF-8 which seems unnecessary but the “//IGNORE” part is important – that means it will throw out any characters it can’t represent appropriately. This was added to prevent a proof of concept attack that exploited a C0 ASCII control code character to break the output (discovered on the sla.ckers.org forums).

The next function, htmlspecialchars(), is a pretty well known function and converts special characters to their ASCII equivalents. The second parameter specifically asks it to encode single quotes.

Lastly we use the array of patterns and replacements declared at the beginning to encode a few final symbols, like parenthesis and the percentage sign, into HTML entities.

This system has worked fairly well for a few years now and as issues are discovered we make changes to it. If you’re looking for the latest code please be sure to check our repository. And, as always, if you find any kind of exploit on AMO please let me know!

The problem we’re running into in the bug linked above is knowing when to expire a cache. Consider when an add-on author updates the summary of their add-on. We know we’ll have to flush the queries on that page out of memcached, and that’s easy enough, but what about all the other places the summary is used? Search results pages, add-on detail pages, recommended lists, the API, etc. Now we’ve got to figure out the queries used on those pages and expire them too. Suddenly I’m wishing we were caching objects in memcached instead of queries.

I looked in to other ways to use memcached and they all have their pros and cons. Caching entire pages means we’d have to store different versions for a person that is logged in vs. logged out and also what permissions they had (pages have different options for localizers, admins, developers, etc.). Caching objects is attractive, but the way CakePHP does queries makes this a non-option (namely, it’s not asking objects for values, it does joins directly on the db). Directly caching queries seems like the best fit because we can affect just the parts of the pages we want and it will work with CakePHP’s current system…just as soon as we figure out how to relate updating a row to all of it’s associated queries.

I attached an idea to the bug but regardless of the process we use, figuring out how to implement a full time cache that we can expire on the fly is going to be an important step in keeping the AMO site usable as our traffic increases.

I downloaded 1.1.15 and 1.1.19, set up a couple symlinks to swap them into my dev copy and looked at how hard it would be to upgrade.

Hats off to the CakePHP developers. After merging in a short patch to the core CakePHP session code, most of the site worked right out of the box. I made a few minor tweaks to our code for things that had changed (and filed ticket 4140 for them) but all in all it was pretty painless. Nice work guys. It shows a lot of planning and foresight to make a framework that doesn’t have a bunch of code-breaking changes after years of development.

First, some background:

The uniqueness of a cookie in the browser is determined by all the attributes when it’s set (not just it’s name). That means I can have multiple cookies named AMOv3 as long as another attribute (eg. path) is different when I set the second cookie.

According to RFC 2109:

If multiple cookies satisfy the criteria [...] they are ordered in the Cookie header such that those with more specific Path attributes precede those with less specific.

In PHP, however, cookies are indexed in the $_COOKIE variable by name, which means I can send several cookies with the same name and only the first cookie will show up. ^[1]

So, what’s this have to do with AMO? For some reason, CakePHP hasn’t been consistent in the past regarding cookie paths. Sometimes they were / (which is correct), sometimes they were something else like /en-US/firefox/users/ and sometimes users had multiple cookies with different paths.

From the PHP manual:

Cookies must be deleted with the same parameters as they were set with. If the value argument is an empty string, or FALSE, and all other arguments match a previous call to setcookie, then the cookie with the specified name will be deleted from the remote client.

When a browser sends a cookie back to a server it only sends the name and value. If a cookie was set with a path that I don’t know I have no way to remove that cookie!

To compound the problem, if a cookie with the same name has a more specific path, it shows up in $_COOKIE and there is no indication the other cookie even exists. This means on the front page your session can be fine and after clicking on a deeper URL you’re suddenly requesting a session that has been expired long ago.

We rolled out a change to session handling last week that prevented myself and another developer from logging in because we had a mess of cookies with different paths. I haven’t had a barrage of emails so I don’t think it’s affected many people, but if you have any troubles with logging in please let me know. If you’re in a hurry, clearing your cookies for the addons.mozilla.org domain will fix any problems.

For the record, any new login cookies on AMO will expire when the browser closes which should prevent stale cookies from coming back to haunt us.

[1] If you need to get at all the cookies the browser is sending, you’ll have to dig through $_SERVER['HTTP_COOKIE'] manually.

Turns out CakePHP considers a TINYINT(1) to be a Boolean. Judging from all the support tickets that have been filed, I’m not the first person to get taken off guard by this behavior. When I asked about it on IRC, the response was that since MySQL considers a TINYINT(1) to be a Boolean, CakePHP does too. That’s not true.

From the MySQL manual:

As of MySQL 5.0.3, a BIT data type is available for storing bit-field values. (Before 5.0.3, MySQL interprets BIT as TINYINT(1).)

That’s saying if I request a BIT it will make it a TINYINT, not if I request a TINYINT it will make it a BIT. Having a framework change the definitions of my database columns sounds crazy to me, but judging from the ticket filed in 2006 this has been CakePHP’s policy for a long time. Despite the long standing precedent I can’t find any documentation about it online other than the closing remarks of those support tickets.

Today’s subject is anonymous sessions. Frameworks (and developers) love them because they are simple and convenient, but it comes at a cost. Keeping track of sessions for every visitor on a high traffic site is far too expensive to be practical. Developers should know how to work around this, but their frameworks need to support them.

The first framework on my mind is Drupal. I filed an issue last year that Drupal should support disabling anonymous sessions. It’s still unassigned so I’m guessing it’s not a high priority, but it was one of the main things that made me choose not to use Drupal on mozilla.com. I wrote some ideas on how to handle it and got some responses from people suffering the same fate. No word on any progress though.

The second framework, CakePHP, has an AUTO_SESSION variable that, just like $cacheQueries, is far to easy to misplace faith in.

By setting AUTO_SESSION to false, you can’t read or write to the session. Working as advertised? Not so much. If you take a closer look at what’s actually happening you’ll see that the session is still getting started, it’s just that CakePHP is blocking your access to it. Even with AUTO_SESSION off, a cookie with a unique ID is set, and a row is still inserted into the sessions table. That last part almost brought down AMO last night. I wrote a patch that disables anonymous sessions for real, but anyone that has talked to me about patching core code knows I don’t like to do it.

When you’re writing code, framework or not, don’t forget about scalability.

Cake’s models have a Boolean variable called $cacheQueries which I misplaced my faith in early on. Turns out this does disable query caching…sometimes.

The test I was writing was pretty straight forward: Look in the database for some information to make sure it wasn’t there, add the info, look in the database to make sure it was there. You can see the actual code, but this example is simplified:

1. $this->Addon->cacheQueries = false; // Disable the built-in caching.
2. $ret = $this->Addon->query("SELECT addon_id FROM `addons_tags` WHERE addon_id={$this->testdata['addonid']}");
3. $this->assertEmpty($ret, 'Data exists!'); // this should be empty
4. $this->Addon->doSomething($this->testdata['addonid']);
5. $ret = $this->Addon->query("SELECT addon_id FROM `addons_tags` WHERE addon_id={$this->testdata['addonid']}");
6. $this->assertNotEmpty($ret, 'Data exists!'); // The array should have info in it

The problem is, line 5 always returned an empty array. I turned the DEBUG mode to 2 so Cake would print out all the queries it was doing and discovered it never actually made the second SELECT call. I smell query caching!

Let’s dig through some CakePHP code to figure out what’s up:

$this->Addon->query() is passed through AppModel::query() to Model::query() and eventually works it’s way down to DboSource::query() with the same arguments. This is where things go south. Near the top of that function you’ll see:

    if (count($args) == 1) {
        return $this->fetchAll($args[0]);

The function signature for DboSource::fetchAll() looks like:

    function fetchAll($sql, $cache = true, $modelName = null)

You can see the second parameter is a Boolean for caching and by default it’s on. That’s the fly in my clam chowder!

So, two ways around it:

The first way is to scroll up about 20 lines and look at the end of DboSource::query(). There is a handler there for a second parameter to your original query() function. Pass in false and voila, things just work. Of course, I didn’t realize this until I was writing this post. I’m pretty sure it’s not documented anywhere unless you look at the code.

The second way is to use execute() instead of query(). If you look at DboSource::execute() you’ll see it suffers no caching and is as close to a direct SQL call as you can get.

The CakePHP manual has this to say about query() and execute():

Custom SQL calls can be made using the model’s query() and execute() methods. The difference between the two is that query() is used to make custom SQL queries (the results of which are returned), and execute() is used to make custom SQL commands (which require no return value).

Apparently there are some other differences as well. (The fact that execute() returns results seems counter to what the manual suggests in the paragraph above but I may just be misinterpreting what they mean.) Regardless, if your tests are failing double check that you’re not fighting the cache and save yourself some time.