Skip to content

New AMO release cycle

I’ve been the lead of the AMO 3.4.x development release cycles which have been going on for about 8 weeks. During that time we’ve been a little more structured with our release schedule. The general plan has been a code freeze every other Monday at midnight and a push to production the following Thursday. This gives us roughly ten days of coding and two days of QA and L10n for each push. It’s a compromise but it means that we’re never too far away from bug fixes and new features. The short turn around cycle also means there aren’t dramatic changes which lead to confusion and panic (I’m looking at you mozillaZine ;) <3 Heart).

In an effort to keep the community informed I’ve also posted to the Webdev blog when each release is going out and linked to the bugs that are being fixed.

We’ve talked internally and think the new release process is working well despite some cramming that consistently happens right before the code freeze (also the FF3 launch threw off our schedule a bit). I’m posting this here to ask if anyone has noticed the change or if there is anything else anyone would like to see from our release process (aside from fixing all the bugs…). Let us know how we can improve the process and we’ll keep working on the bug list.

Tagged ,
2008 06 25 Wil Clouser Comments (1) Permalink

Speed testing your web site with AOL Pagetest

I’m at Velocity 2008 right now and the keynote I was most excited about this morning introduced http://webpagetest.org/.

This site accepts a URL which it loads over a network connection with the speed of your choice. After loading the site it offers you a waterfall graph of page elements, a screenshot so you can see if it loaded correctly, and automatic comparisons to previous page loads to see how caching affects your site.

The code behind the page is based on the open source AOL Pagetest which means it can be modified as needed. I think I heard Ryan say something about automated testing over time with historical graphs and email alerts? :)

Tagged , ,
2008 06 24 Wil Clouser Comments (2) Permalink

addons.mozilla.org and Localization at the Firefox Summit

We’re planning several sessions at the upcoming Firefox summit including one specifically about AMO and L10n in which we hope to connect with localizers and start looking at some of the pain points of localizing AMO. With that in mind I’ve been collecting topic ideas and I’d like to hear any others you have. On my list so far:

  • Connecting localizers and add-on authors. The only way to localize add-ons and metadata right now is if the author knows someone who can translate the extension or by using an existing service like BabelZilla. It would be nice to add an easy way to connect these two without overwhelming localizers or creating a bottleneck.
  • Being notified of AMO string updates. Right now we have RSS feeds and the mailing list. It would be nice to have something more concrete and consistently up to date.
  • Use normal .po files or make the ones we have easier to use. This came up before and was discussed again briefly. I don’t expect to convert to using standard .po files now that we can provide English fallback in the application. However, we can make them easier to use particularly with the small (size and number) of updates for AMO.

What other ideas and pain points would you like to discuss at the summit? Leave a comment with ideas.

Also, I expect Verbatim to come up in our discussions since it’s still in planning and now is the time to add features. If you have ideas for AMO in a Verbatim context feel free to add them also.

Tagged , , ,
2008 06 02 Wil Clouser Comments (1) Permalink

Making Life Easier for Localizers - Introducing Verbatim

The webdev and L10n-drivers teams have been talking about implementing an online localization tool for a few months and I’m happy to report that a plan is progressing. Seth Bindernagel has been giving updates about the planning process and the direction we’re going in. Our planning decisions will probably come up but my focus here will be on the actual implementation of the tool.

We’ve decided to use the codename Verbatim to refer to this project.

I’ve started some wiki pages that are short but growing. As usual there is a meeting notepad which anyone is welcome to peruse. The wiki page offers a brief description but I can give a short summary of our current situation and where I’d like to see it go. This tool will be used for both the Firefox browser and Mozilla web sites. I expect my posts to lean towards talking about the web since that’s where my brain is but it’s important to keep both in mind.

The current situation:

  • L10n projects are scattered around in different places. Most web sites are in SVN but the browser’s L10n is in CVS.
  • L10n projects use several different formats.
  • There are unnecessarily complex permissions in our system. Localizers need an LDAP account and then have to be specifically granted permission in the SVN directory to commit. Sometimes those two get decoupled and strange permission errors happen.
  • Some of our projects use a potentially slow process. For example, on AMO, some localizers still add updated translations to a bug as attachments and then I commit them to SVN manually. This takes time and if I don’t get to it for a while the localizer is stuck waiting on me to update before they can verify the changes.
  • When we add or change a string on a web site we generally just email the dev-l10n-web mailing list which is easy to miss. There is an RSS feed coming out of SVN but that’s still not an ideal notification system.

The plan:

  • Provide a central point of localization where someone can see all L10n projects for a locale and the amount of work to be done on each.
  • Provide a common interface for localizing all of our different file types.
  • Provide an easy way to change the strings online[1].
  • Provide an easy way to be notified of new and changed strings.
  • Optionally, provide the public with a way to suggest changes to existing strings. L10n leaders will be able to approve or reject these suggestions at their convenience.

After reviewing many tools online, in the interest of time, completeness, and alignment with the plan, the webdev and L10n-drivers team have decided to use Pootle and the Translate Toolkit for our initial release. Pootle in it’s current form already implements the basics of Firefox L10n and supports regular .po files. The webdev goal over the next few weeks will be to setup an official Pootle development area and start figuring out how to convert our L10n projects’ file types to something Pootle can import and export.

I’m hoping these changes will encourage new users to get involved and make things easier for our long time contributers. The plan is still in development and, as usual, input is welcome. Real time discussion is easiest in #verbatim on IRC. Email and comments here are effective as well.

[1] Just to be clear, I don’t want this tool to become a bottleneck or hinder localizers that have a good process down. It will still be possible to do check-ins to the repositories manually without using the web interface. The current Pootle web interface can be played with or you can watch a screencast for more details.

Tagged , ,
2008 05 28 Wil Clouser Comments (3) Permalink

Mother’s day pie

Two Pies

Not that I need a special occasion to bake a tasty pie but mother’s day combined with picking 6 pounds of fresh rhubarb added up to a good excuse.

I puttered around on the internet looking for less ordinary recipes than what I was used to and came up with an idea and a recipe.

The pie with the lattice top was a regular strawberry/rhubarb mix but I stirred the fruit in a custard before adding it which made it richer with a hint of nutmeg.

The pie in the background was a Rhubarb and Pineapple Pie. There were four layers of fruit in the pie: two pineapple and two rhubarb. I thought it was a good combination, but really, we’re talking about butter, sugar, and fruit - how bad is it going to be? You could put pomegranates and kiwis in there and it’d probably still taste great.

Oh, one more thing - what’s the deal with tiny pies? That recipe I linked to calls for two cups of rhubarb and one cup of pineapple…really? That’s it? If I’m going to all the trouble to make pie crusts and bake a pie I’m going to fill that bad boy up. I put in around 5 cups of rhubarb and an entire pineapple. Of course, that meant a piece of the pie wasn’t so much dessert as the whole meal.

Tagged
2008 05 15 Wil Clouser Comments (2) Permalink

Memcached best practices and internals

Yesterday, igvita.com published a short article summarizing memcached internals and best practices. It was originally a talk by Brian Aker and Alan Kasindorf and their slides are included. It’s a good read and digs into memcached’s memory management and expiration logic. (Thanks for the heads up, chenb).

I noticed their first best practice for memcached, “Don’t think row-level (database) caching, think complex objects, ” is in opposition with what we’re doing on AMO. After stuff like bug 425315 I completely agree with them.

Tagged , , ,
2008 04 23 Wil Clouser Comments (0) Permalink

Caching is easy; Expiration is hard

Still on a high from our success with memcached in AMO version 2, we decided to go a fairly common route and cache query results in version 3. This performs admirably particularly with our rediculously long and slow queries. Over time, though, the popularity of the site and the load on the servers climb, and soon we’re looking at slowness issues again. On a rough day we decided to increase the expiration timeout for our queries in memcached from a minute to around an hour. This gives the database servers some breathing room but causes excessive delay on the AMO site when add-ons are updated and things like bug 425315 and it’s friends are born. Weird things happen when parts of a site expire at different times and consequently user experience (particularly add-on developers) suffers.

The problem we’re running into in the bug linked above is knowing when to expire a cache. Consider when an add-on author updates the summary of their add-on. We know we’ll have to flush the queries on that page out of memcached, and that’s easy enough, but what about all the other places the summary is used? Search results pages, add-on detail pages, recommended lists, the API, etc. Now we’ve got to figure out the queries used on those pages and expire them too. Suddenly I’m wishing we were caching objects in memcached instead of queries.

I looked in to other ways to use memcached and they all have their pros and cons. Caching entire pages means we’d have to store different versions for a person that is logged in vs. logged out and also what permissions they had (pages have different options for localizers, admins, developers, etc.). Caching objects is attractive, but the way CakePHP does queries makes this a non-option (namely, it’s not asking objects for values, it does joins directly on the db). Directly caching queries seems like the best fit because we can affect just the parts of the pages we want and it will work with CakePHP’s current system…just as soon as we figure out how to relate updating a row to all of it’s associated queries.

I attached an idea to the bug but regardless of the process we use, figuring out how to implement a full time cache that we can expire on the fly is going to be an important step in keeping the AMO site usable as our traffic increases.

Tagged , , , ,
2008 04 23 Wil Clouser Comments (5) Permalink

AMO Scalability: Then and Now

Struggling with scalability on AMO is nothing new but the tools we use to solve the problems have changed over time. Here is a bit of information on the performance evolution AMO has gone through. I wanted to link to the wayback machine for all our old versions, but I get “Redirect Errors” for the addons.mozilla.org domain. I’ll have to make due with code repositories.

Version 1 of AMO wasn’t concerned with caching. It was straight PHP talking directly to a single MySQL box. Short, easy, and not very scalable.

Version 2 of AMO progressed through several caching systems. The site used the Smarty template engine so our first step was to turn on the built in Smarty cache. That didn’t give us the performance we needed, so Mike Morgan started caching page output in PEAR’s Cache_Lite. I don’t remember the specifics of this implementation since it was so short lived (less than a month), but the CVS log, mentions problems with “scalability in a clustered environment.” Our next step was to store the same page output in memcached instead of Cache_Lite which brought pretty satisfying results. Thus began our abuse of memcached.

In addition to memcached and expanding the number of web servers it ran on, version 2 also boasted two other significant performance improvements. The first was the ability to talk to a slave database for read-only queries which, when combined with a load balancer, let us scale database servers horizontally. The second was installing a NetScaler in front of addons.mozilla.org giving us the benefits of a reverse proxy cache and SSL offloading. These changes bought us precious time when hoards of Firefox 1.5 users were clamoring for add-ons. In fact, I’d say we were in pretty good shape at that point.

Fast forward to Version 3 (the current version). We’ve expanded the memcache servers from one to two and instead of page output we’re storing database queries and their results. We’re still using a single master database but are using two slaves now for read only queries. There are several NetScalers around the world caching pages locally[1] for closer regions. We’ve survived quite a while on this system but we’re starting to push the envelope again and we’re going to need to make some changes to be able to scale for Firefox 3 and still provide a good user experience. I’ll write more about our plans as they develop.

[1] Users who are logged in to AMO don’t get the local caches - their connection is always to San Jose, CA.

Tagged , , , , ,
2008 04 18 Wil Clouser Comments (6) Permalink

Another warning option for submitting forms?

These are our current options for submitting forms in Firefox 3:
Firefox security options dialog

I don’t know anyone that has the “I submit information that’s not encrypted” option checked. We used to prompt for submitting information that was unencrypted, next we added an option on the dialog that disabled the warning (and was checked by default), and finally we removed the warning by default all together. People submit so much information via searches, surveys, voting, sending quick messages, etc. that it would just get in the way and be ignored.

I’ve noticed lately that a lot of sites are showing login forms on unencrypted pages but submitting them via SSL to their target pages. This is a secure method of logging in but it’s started to train me to not look for a secure connection before I log in and that’s not a good idea in the long run.

I think our current option is too broad, so I’m proposing that we add an option that says “I submit credentials that aren’t encrypted” or a sub-option for the current text that says something about “unless I’m sending a password.”

In more technical language: If a user POSTs a form containing an input of type=”password” to a non-encrypted page we should show a warning.

This would mean regular searches, filling in surveys, anonymous voting and polls would all pass transparently if unencrypted, but when you got to a form with a password you would be warned.

I bounced this idea off a channel in IRC and no one said I was nuts so I figured I’d take it here. It seems like too small of a feature to make an add-on out of but I might if I get some free time. What do you think?

Tagged , ,
2008 04 04 Wil Clouser Comments (19) Permalink

CakePHP makes upgrading easy

Laura attended CakeFest a couple months ago and got to meet some core Cake developers in person. In doing so she let slip that AMO was running on a pretty old version (1.1.12 - Released in December of 2006). Apparently 1.1.15 had some major performance boosts and since we melted the cluster a few times recently (the new API was the culprit) we thought it would be a good idea to investigate upgrading.

I downloaded 1.1.15 and 1.1.19, set up a couple symlinks to swap them into my dev copy and looked at how hard it would be to upgrade.

Hats off to the CakePHP developers. After merging in a short patch to the core CakePHP session code, most of the site worked right out of the box. I made a few minor tweaks to our code for things that had changed (and filed ticket 4140 for them) but all in all it was pretty painless. Nice work guys. It shows a lot of planning and foresight to make a framework that doesn’t have a bunch of code-breaking changes after years of development.

Tagged , , ,
2008 04 01 Wil Clouser Comments (0) Permalink