Version 1 of AMO wasn’t concerned with caching. It was straight PHP talking directly to a single MySQL box. Short, easy, and not very scalable.

Version 2 of AMO progressed through several caching systems. The site used the Smarty template engine so our first step was to turn on the built in Smarty cache. That didn’t give us the performance we needed, so Mike Morgan started caching page output in PEAR’s Cache_Lite. I don’t remember the specifics of this implementation since it was so short lived (less than a month), but the CVS log, mentions problems with “scalability in a clustered environment.” Our next step was to store the same page output in memcached instead of Cache_Lite which brought pretty satisfying results. Thus began our abuse of memcached.

In addition to memcached and expanding the number of web servers it ran on, version 2 also boasted two other significant performance improvements. The first was the ability to talk to a slave database for read-only queries which, when combined with a load balancer, let us scale database servers horizontally. The second was installing a NetScaler in front of addons.mozilla.org giving us the benefits of a reverse proxy cache and SSL offloading. These changes bought us precious time when hoards of Firefox 1.5 users were clamoring for add-ons. In fact, I’d say we were in pretty good shape at that point.

Fast forward to Version 3 (the current version). We’ve expanded the memcache servers from one to two and instead of page output we’re storing database queries and their results. We’re still using a single master database but are using two slaves now for read only queries. There are several NetScalers around the world caching pages locally[1] for closer regions. We’ve survived quite a while on this system but we’re starting to push the envelope again and we’re going to need to make some changes to be able to scale for Firefox 3 and still provide a good user experience. I’ll write more about our plans as they develop.

[1] Users who are logged in to AMO don’t get the local caches – their connection is always to San Jose, CA.

Turns out CakePHP considers a TINYINT(1) to be a Boolean. Judging from all the support tickets that have been filed, I’m not the first person to get taken off guard by this behavior. When I asked about it on IRC, the response was that since MySQL considers a TINYINT(1) to be a Boolean, CakePHP does too. That’s not true.

From the MySQL manual:

As of MySQL 5.0.3, a BIT data type is available for storing bit-field values. (Before 5.0.3, MySQL interprets BIT as TINYINT(1).)

That’s saying if I request a BIT it will make it a TINYINT, not if I request a TINYINT it will make it a BIT. Having a framework change the definitions of my database columns sounds crazy to me, but judging from the ticket filed in 2006 this has been CakePHP’s policy for a long time. Despite the long standing precedent I can’t find any documentation about it online other than the closing remarks of those support tickets.

mysql> select averagerating from addons where id=5001;
+---------------+
| averagerating |
+---------------+
| 0             |
+---------------+
1 row in set (0.00 sec)

mysql> UPDATE addons SET averagerating = 9.33 WHERE id = 5001;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select averagerating from addons where id=5001;
+-----------------------------------------------------+
| averagerating                                       |
+-----------------------------------------------------+
| 9.3300000000000000710542735760100185871124267578125 |
+-----------------------------------------------------+
1 row in set (0.00 sec)

mysql> UPDATE addons SET averagerating = '9.33' WHERE id = 5001;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select averagerating from addons where id=5001;
+---------------+
| averagerating |
+---------------+
| 9.33          |
+---------------+
1 row in set (0.00 sec)

Out of 1131 updates, 67 of them ended up with the extra characters and the rest looked just fine. That means if you don’t double check the data pretty closely, something like this could slip by and lead to trouble later on. Double check your SQL before you run it!