The webdev and L10n-drivers teams have been talking about implementing an online localization tool for a few months and I’m happy to report that a plan is progressing. Seth Bindernagel has been giving updates about the planning process and the direction we’re going in. Our planning decisions will probably come up but my focus here will be on the actual implementation of the tool.
Yesterday, igvita.com published a short article summarizing memcached internals and best practices. It was originally a talk by Brian Aker and Alan Kasindorf and their slides are included. It's a good read and digs into memcached's memory management and expiration logic. (Thanks for the heads up, chenb).
Still on a high from our success with memcached in AMO version 2, we decided to go a fairly common route and cache query results in version 3. This performs admirably particularly with our ridiculously long and slow queries. Over time, though, the popularity of the site and the load on the servers climb, and soon we’re looking at slowness issues again. On a rough day we decided to increase the expiration timeout for our queries in memcached from a minute to around an hour. This gives the database servers some breathing room but causes excessive delay on the AMO site when add-ons are updated and things like bug 425315 and it’s friends are born. Weird things happen when parts of a site expire at different times and consequently user experience (particularly add-on developers) suffers.
Struggling with scalability on AMO is nothing new but the tools we use to solve the problems have changed over time. Here is a bit of information on the performance evolution AMO has gone through. I wanted to link to the wayback machine for all our old versions, but I get "Redirect Errors" for the addons.mozilla.org domain. I'll have to make due with code repositories.
These are our current options for submitting forms in Firefox 3: