Skip to content

Mother’s day pie

Two Pies

Not that I need a special occasion to bake a tasty pie but mother’s day combined with picking 6 pounds of fresh rhubarb added up to a good excuse.

I puttered around on the internet looking for less ordinary recipes than what I was used to and came up with an idea and a recipe.

The pie with the lattice top was a regular strawberry/rhubarb mix but I stirred the fruit in a custard before adding it which made it richer with a hint of nutmeg.

The pie in the background was a Rhubarb and Pineapple Pie. There were four layers of fruit in the pie: two pineapple and two rhubarb. I thought it was a good combination, but really, we’re talking about butter, sugar, and fruit - how bad is it going to be? You could put pomegranates and kiwis in there and it’d probably still taste great.

Oh, one more thing - what’s the deal with tiny pies? That recipe I linked to calls for two cups of rhubarb and one cup of pineapple…really? That’s it? If I’m going to all the trouble to make pie crusts and bake a pie I’m going to fill that bad boy up. I put in around 5 cups of rhubarb and an entire pineapple. Of course, that meant a piece of the pie wasn’t so much dessert as the whole meal.

Tagged
2008 05 15 Wil Clouser Comments (1) Permalink

Memcached best practices and internals

Yesterday, igvita.com published a short article summarizing memcached internals and best practices. It was originally a talk by Brian Aker and Alan Kasindorf and their slides are included. It’s a good read and digs into memcached’s memory management and expiration logic. (Thanks for the heads up, chenb).

I noticed their first best practice for memcached, “Don’t think row-level (database) caching, think complex objects, ” is in opposition with what we’re doing on AMO. After stuff like bug 425315 I completely agree with them.

Tagged , , ,
2008 04 23 Wil Clouser Comments (0) Permalink

Caching is easy; Expiration is hard

Still on a high from our success with memcached in AMO version 2, we decided to go a fairly common route and cache query results in version 3. This performs admirably particularly with our rediculously long and slow queries. Over time, though, the popularity of the site and the load on the servers climb, and soon we’re looking at slowness issues again. On a rough day we decided to increase the expiration timeout for our queries in memcached from a minute to around an hour. This gives the database servers some breathing room but causes excessive delay on the AMO site when add-ons are updated and things like bug 425315 and it’s friends are born. Weird things happen when parts of a site expire at different times and consequently user experience (particularly add-on developers) suffers.

The problem we’re running into in the bug linked above is knowing when to expire a cache. Consider when an add-on author updates the summary of their add-on. We know we’ll have to flush the queries on that page out of memcached, and that’s easy enough, but what about all the other places the summary is used? Search results pages, add-on detail pages, recommended lists, the API, etc. Now we’ve got to figure out the queries used on those pages and expire them too. Suddenly I’m wishing we were caching objects in memcached instead of queries.

I looked in to other ways to use memcached and they all have their pros and cons. Caching entire pages means we’d have to store different versions for a person that is logged in vs. logged out and also what permissions they had (pages have different options for localizers, admins, developers, etc.). Caching objects is attractive, but the way CakePHP does queries makes this a non-option (namely, it’s not asking objects for values, it does joins directly on the db). Directly caching queries seems like the best fit because we can affect just the parts of the pages we want and it will work with CakePHP’s current system…just as soon as we figure out how to relate updating a row to all of it’s associated queries.

I attached an idea to the bug but regardless of the process we use, figuring out how to implement a full time cache that we can expire on the fly is going to be an important step in keeping the AMO site usable as our traffic increases.

Tagged , , , ,
2008 04 23 Wil Clouser Comments (5) Permalink

AMO Scalability: Then and Now

Struggling with scalability on AMO is nothing new but the tools we use to solve the problems have changed over time. Here is a bit of information on the performance evolution AMO has gone through. I wanted to link to the wayback machine for all our old versions, but I get “Redirect Errors” for the addons.mozilla.org domain. I’ll have to make due with code repositories.

Version 1 of AMO wasn’t concerned with caching. It was straight PHP talking directly to a single MySQL box. Short, easy, and not very scalable.

Version 2 of AMO progressed through several caching systems. The site used the Smarty template engine so our first step was to turn on the built in Smarty cache. That didn’t give us the performance we needed, so Mike Morgan started caching page output in PEAR’s Cache_Lite. I don’t remember the specifics of this implementation since it was so short lived (less than a month), but the CVS log, mentions problems with “scalability in a clustered environment.” Our next step was to store the same page output in memcached instead of Cache_Lite which brought pretty satisfying results. Thus began our abuse of memcached.

In addition to memcached and expanding the number of web servers it ran on, version 2 also boasted two other significant performance improvements. The first was the ability to talk to a slave database for read-only queries which, when combined with a load balancer, let us scale database servers horizontally. The second was installing a NetScaler in front of addons.mozilla.org giving us the benefits of a reverse proxy cache and SSL offloading. These changes bought us precious time when hoards of Firefox 1.5 users were clamoring for add-ons. In fact, I’d say we were in pretty good shape at that point.

Fast forward to Version 3 (the current version). We’ve expanded the memcache servers from one to two and instead of page output we’re storing database queries and their results. We’re still using a single master database but are using two slaves now for read only queries. There are several NetScalers around the world caching pages locally[1] for closer regions. We’ve survived quite a while on this system but we’re starting to push the envelope again and we’re going to need to make some changes to be able to scale for Firefox 3 and still provide a good user experience. I’ll write more about our plans as they develop.

[1] Users who are logged in to AMO don’t get the local caches - their connection is always to San Jose, CA.

Tagged , , , , ,
2008 04 18 Wil Clouser Comments (6) Permalink

Another warning option for submitting forms?

These are our current options for submitting forms in Firefox 3:
Firefox security options dialog

I don’t know anyone that has the “I submit information that’s not encrypted” option checked. We used to prompt for submitting information that was unencrypted, next we added an option on the dialog that disabled the warning (and was checked by default), and finally we removed the warning by default all together. People submit so much information via searches, surveys, voting, sending quick messages, etc. that it would just get in the way and be ignored.

I’ve noticed lately that a lot of sites are showing login forms on unencrypted pages but submitting them via SSL to their target pages. This is a secure method of logging in but it’s started to train me to not look for a secure connection before I log in and that’s not a good idea in the long run.

I think our current option is too broad, so I’m proposing that we add an option that says “I submit credentials that aren’t encrypted” or a sub-option for the current text that says something about “unless I’m sending a password.”

In more technical language: If a user POSTs a form containing an input of type=”password” to a non-encrypted page we should show a warning.

This would mean regular searches, filling in surveys, anonymous voting and polls would all pass transparently if unencrypted, but when you got to a form with a password you would be warned.

I bounced this idea off a channel in IRC and no one said I was nuts so I figured I’d take it here. It seems like too small of a feature to make an add-on out of but I might if I get some free time. What do you think?

Tagged , ,
2008 04 04 Wil Clouser Comments (19) Permalink

CakePHP makes upgrading easy

Laura attended CakeFest a couple months ago and got to meet some core Cake developers in person. In doing so she let slip that AMO was running on a pretty old version (1.1.12 - Released in December of 2006). Apparently 1.1.15 had some major performance boosts and since we melted the cluster a few times recently (the new API was the culprit) we thought it would be a good idea to investigate upgrading.

I downloaded 1.1.15 and 1.1.19, set up a couple symlinks to swap them into my dev copy and looked at how hard it would be to upgrade.

Hats off to the CakePHP developers. After merging in a short patch to the core CakePHP session code, most of the site worked right out of the box. I made a few minor tweaks to our code for things that had changed (and filed ticket 4140 for them) but all in all it was pretty painless. Nice work guys. It shows a lot of planning and foresight to make a framework that doesn’t have a bunch of code-breaking changes after years of development.

Tagged , , ,
2008 04 01 Wil Clouser Comments (0) Permalink

My Homemade Vanilla Extract

Bottled vanilla extract
I haven’t found much time for cooking lately but this project brought me back to the kitchen tonight. I’ve been acquiring the ingredients for about a month and finally found a suitable bottle (it was originally a bottle of Mönchshof World Classic Schwarz). I followed these instructions for the most part but I added a tablespoon of dark rum for additional flavor.

The site claims the vanilla can be used as soon as a month but I expect to filter it around September and then transfer it to smaller bottles. All in all I should get about a cup and a half of extract. I hope it’s as delicious as it smells. :)

Tagged
2008 03 15 Wil Clouser Comments (2) Permalink

How my cookies became a one way street

I’ve been playing with CakePHP’s session code lately and ran across an interesting (read: nerdy) problem with cookies on AMO.

First, some background:

The uniqueness of a cookie in the browser is determined by all the attributes when it’s set (not just it’s name). That means I can have multiple cookies named AMOv3 as long as another attribute (eg. path) is different when I set the second cookie.

According to RFC 2109:

If multiple cookies satisfy the criteria […] they are ordered in the Cookie header such that those with more specific Path attributes precede those with less specific.

In PHP, however, cookies are indexed in the $_COOKIE variable by name, which means I can send several cookies with the same name and only the first cookie will show up. [1]

So, what’s this have to do with AMO? For some reason, CakePHP hasn’t been consistent in the past regarding cookie paths. Sometimes they were / (which is correct), sometimes they were something else like /en-US/firefox/users/ and sometimes users had multiple cookies with different paths.

From the PHP manual:

Cookies must be deleted with the same parameters as they were set with. If the value argument is an empty string, or FALSE, and all other arguments match a previous call to setcookie, then the cookie with the specified name will be deleted from the remote client.

When a browser sends a cookie back to a server it only sends the name and value. If a cookie was set with a path that I don’t know I have no way to remove that cookie!

To compound the problem, if a cookie with the same name has a more specific path, it shows up in $_COOKIE and there is no indication the other cookie even exists. This means on the front page your session can be fine and after clicking on a deeper URL you’re suddenly requesting a session that has been expired long ago.

We rolled out a change to session handling last week that prevented myself and another developer from logging in because we had a mess of cookies with different paths. I haven’t had a barrage of emails so I don’t think it’s affected many people, but if you have any troubles with logging in please let me know. If you’re in a hurry, clearing your cookies for the addons.mozilla.org domain will fix any problems.

For the record, any new login cookies on AMO will expire when the browser closes which should prevent stale cookies from coming back to haunt us.

[1] If you need to get at all the cookies the browser is sending, you’ll have to dig through $_SERVER[’HTTP_COOKIE’] manually.

Tagged , , ,
2008 03 11 Wil Clouser Comments (0) Permalink

When is a TINYINT(1) not a TINYINT(1)?

When you’re using CakePHP!

Turns out CakePHP considers a TINYINT(1) to be a Boolean. Judging from all the support tickets that have been filed, I’m not the first person to get taken off guard by this behavior. When I asked about it on IRC, the response was that since MySQL considers a TINYINT(1) to be a Boolean, CakePHP does too. That’s not true.

From the MySQL manual:

As of MySQL 5.0.3, a BIT data type is available for storing bit-field values. (Before 5.0.3, MySQL interprets BIT as TINYINT(1).)

That’s saying if I request a BIT it will make it a TINYINT, not if I request a TINYINT it will make it a BIT. Having a framework change the definitions of my database columns sounds crazy to me, but judging from the ticket filed in 2006 this has been CakePHP’s policy for a long time. Despite the long standing precedent I can’t find any documentation about it online other than the closing remarks of those support tickets.

Tagged , , ,
2008 03 07 Wil Clouser Comments (2) Permalink

Frameworks that start sessions for every visitor make me sad

I might have played the devil’s advocate when Lars was hating on frameworks at the barcamp last weekend, but that doesn’t mean I don’t see his point. The latest in a series of frustrations with frameworks kept me up until 3am last night. What better way to cap it off than complaining on the internet?

Today’s subject is anonymous sessions. Frameworks (and developers) love them because they are simple and convenient, but it comes at a cost. Keeping track of sessions for every visitor on a high traffic site is far too expensive to be practical. Developers should know how to work around this, but their frameworks need to support them.

The first framework on my mind is Drupal. I filed an issue last year that Drupal should support disabling anonymous sessions. It’s still unassigned so I’m guessing it’s not a high priority, but it was one of the main things that made me choose not to use Drupal on mozilla.com. I wrote some ideas on how to handle it and got some responses from people suffering the same fate. No word on any progress though.

The second framework, CakePHP, has an AUTO_SESSION variable that, just like $cacheQueries, is far to easy to misplace faith in.

By setting AUTO_SESSION to false, you can’t read or write to the session. Working as advertised? Not so much. If you take a closer look at what’s actually happening you’ll see that the session is still getting started, it’s just that CakePHP is blocking your access to it. Even with AUTO_SESSION off, a cookie with a unique ID is set, and a row is still inserted into the sessions table. That last part almost brought down AMO last night. I wrote a patch that disables anonymous sessions for real, but anyone that has talked to me about patching core code knows I don’t like to do it.

When you’re writing code, framework or not, don’t forget about scalability.

Tagged , , , , ,
2008 03 06 Wil Clouser Comments (2) Permalink