Our main issue was hitting a memory limit or the max execution time. We hit the limits often for a variety of reasons, some legitimate bugs, and some because we tried to hack around things to make the tests run. If we change the limits we affect the tests because they are running within the same environment. There wasn’t really a concept of fixtures then, although it looks like CakePHP has stepped up there. The simple test web runner was hard to use and the mock objects were sometimes a little too mocked and missing some attributes.

All in all it was a heroic effort to get that many tests, but we didn’t maintain it because they were so slow to write and difficult to run. Testing can be a pain to write, sure, but it shouldn’t be a burden like that. Enter Django’s testing suite (built on top of Python’s unittest). It has most of our complaints handled out of the box. It’s very well documented, considers a lot of aspects of testing, supports fixtures, a built-in client, etc. It’s a well thought out framework to build tests on.

We’re being more vigilant about requiring tests this time around, but they also aren’t as frustrating to write. When you write them they actually work and they stay working. Most of what you want is built in already. For example, I wrote the password reset form we needed on AMO in Django. With CakePHP and SimpleTest I’d have no idea how to test that the email was actually working. It’s apparently possible with a SimpleTest add-on and enough code that I have to scroll in my browser. With Django’s test suite the actual code was 5 lines, 3 of which were assertions:

    def test_request_success(self):
        self.client.post('/en-US/firefox/users/pwreset',
                        {'email': self.user.email})

        eq_(len(mail.outbox), 1)
        assert mail.outbox[0].subject.find('Password reset') == 0
        assert mail.outbox[0].body.find('pwreset/%s' % self.uidb36) > 0

With the power of the new test suite we’re once again writing and maintaining our unit tests – currently at around 390 tests and increasing steadily. Plenty of people have written about why unit tests are important so I won’t belabor the point, but I will mention that it’s a great feeling to be able to commit something and be confident it hasn’t affected other parts of the site. It’s almost as good of a feeling when you write your code and a completely different test fails pointing out a case that you didn’t even consider but one that would soak up developer time trying to debug down the road.

Building on a foundation that takes testing seriously is great.

AMO is the first Mozilla Webdev site to use continuous integration, and it’s been a long time coming. With the way it’s currently configured we’ve got code coverage trending, unit test trending, code quality trending, as well as detailed reports for all the above for every single check in.

If anything fails or oversteps a threshold our IRC bot complains and we can get it fixed up quickly. It’s a boon to productivity to know that all the code being checked in is being tested automatically, plus it gives everyone a stable state to compare to.

Thanks to everyone that helped get Hudson going, from the people that write it, to the IT team that keeps it alive, to the webdev team that helped work out the kinks.

We hit some limitations with the NetScaler that we weren’t happy with. Cost aside, it ignored some pretty standard stuff like the HTTP Vary Header. After working around that for years we switched to the horizontally scalable Zeus Traffic Manager (at that time, referred to as ZXTM). We’ve been pleased with our choice and six months ago I wrote a similar PHP library that allows you to connect to Zeus’s API. Time and priorities being what they are, we never implemented it in production.

Finally, the real point to this post, last night I wrote a python library that will expire content from Zeus. We’ll roll this into our migration and waiting on content to expire from Zeus should be a thing of the past.

As always, if you can use the libraries, feel free. They all have READMEs with examples.

First out of the chute was the .po files. It turns out the actual formatting is different between the two languages. PHP uses %1$s for its substitutions, but python uses either named variables like (num)s or integers like {0}. For the record, they both support %s when you don’t need to order the substitutions.
PHP example:
I have %2$s apples and %1$s oranges
Python example:
I have {1} apples and {0} oranges

Since I’ve worked with the Translate Toolkit before, I decided to write a script to convert between the two formats. If you find yourself in the same unfortunate boat as me, behold
phppo2pypo and pypo2phppo to convert between the two types.

Crisis averted, right? Oh, that’s just scratching the surface. Remember how happy I was that PHP finally started supporting msgctxt? Well, Python has had a patch for it since 2008 but no one has bothered to land it. I wrote a new ugettext() and ungettext() that recognizes context in the .po files. To use simply do: from l10n import ugettext as _ at the top of your file.

Along with adding msgctxt support, those two functions also collapse consecutive white space. We’re using Jinja2 with Babel and the i18n extension as our template engine. Jinja2 has a concept of stripping white space from the beginning or end of a string but does nothing about the middle. A paragraph of text in a Jinja2 template would look like:

{% trans -%}Mozilla is providing links to these applications
as a courtesy, and makes no representations regarding the
applications or any information related thereto. Any questions,
complaints or claims regarding the applications must be
directed to the appropriate software vendor.
{%- endtrans %}


That’s a decent looking template, right? Yeah, well, when Babel extracts that, it includes all the line breaks too, giving you something like this. The localizers would revolt if I sent them that, so I added in auto white-space collapsing. Getting Babel to use the new functions means a new extraction script.

At this point, we’re extracting strings from our new code and we can convert between Python and PHP files. All we need now is a Frankenstein mix of xgettext functions to act as glue. Meet the amalgamate script that uses the pypo2php scripts, concatenates the .pot files, and merge updates each locales .po file. After that it’s quick tweaks to the build scripts to create z-messages.po files and we’re done.

So, all that said, the new process for L10n, while we’re in this transitional phase, is:

1. From the PHP code, run locale/extract-po-remora.sh. That pulls everything from all the PHP files, creates locale/r-keys.pot, updates the messages.po file for each locale, and compiles them. Life used to be so simple.
2. From the python code, make sure you’re up to date, then run ./manage.py extract. That will pull everything from the python code and templates and create locale/z-keys.pot.
3. Run ./manage.py amalgamate. That will merge the z-keys.pot into the PHP messages.po files.
4. Localizers can make their changes as usual, and commit back to messages.po.
5. From PHP, locale/copy-to-zamboni.py locale will create z-messages.po files in the Python format. We could skip right to .mo files, but in case something goes wrong I want to see the .po files.
6. Then, like today, locale/compile-mo.sh locale will compile all the .po files.

After all those steps are done, we’ve got duplicate .mo files, aside from formatting, and each application can look at its own .mo to get the strings it needs. All this code is just a big band-aid and there are plenty of things that are more fun than juggling L10n between two applications across two RCSs. But we knew what we were getting in to. I’ll post something more positive later to help justify it.

I can look at them any time, and there are a few on flickr that other people can look at, but what about people that aren’t tech savvy or are really busy and get caught up in the daily grind?

I was thinking, the answer can be a really simple script running from cron, say, weekly. It picks a random photo from a directory and emails it to a group of people. That’s it. The idea is:

• it’s low tech compared to RSS feeds and social networking sites. This “just works” with the tools people (potentially, low-tech people) are already used to using.
• in a similar vein, commenting and discussion are built in if you feel like it.
• it’s focused on the people on the CC list. Sending out an old photo that is relevant just to those people has a lot more effect than one scrolling by on flickr. It’s got a personal touch.
• it’s automatic. You wake up Monday morning, drag yourself to work, and there is a photo from 15 years ago in your inbox and you can laugh about how bad your hair was. Awesome.

So, I thought I would try it and cooked up audreytoo. Basically, you seed a directory with copies of the pictures you want to send out, add the script to a cron job, and it does it’s thing. There are a couple other features in the script that you could look at if you want to get fancier.

I realize there is some irony in using an automated script to say “I’m thinking of you” but as long as I’m on the CC list it’s still true. Anyway, I’m sharing the code in case anyone else can find value in it.

On the SVN server the first thing we did was to create a special Verbatim user that can commit to SVN via SSH using a generated key. We copied this key to the Verbatim host which allowed us to commit as the verbatim user without typing a username or password.

The only thing that was added to the Verbatim code was a patch that Dan Schafer cooked up that sets an SVN revision property, translate:author, to the name of the current user. When the user clicks “commit” this property is set and sent along with the commit.

At this point we could commit from the application but it still goes to the application as the Verbatim user. We used SVN’s hooks to take the next step.

The first script we changed was the pre-revprop-change hook. This controls what special revision properties a user can modify when they commit. Our script adds the ability to modify svn:author and translate:author. Before allowing the modifications the script checks if the user committing is the special verbatim user to prevent anyone from committing as someone else.

Next we added a post-commit script that looks for the translate:author property. If it’s found it will take that value, replace svn:author, and remove translate:author; effectively making whatever was in translate:author the real author. This is a non-versioned change which means there is no commit that needs to happen – the new author is set immediately.

With these scripts in place we can commit as anyone from the application and everyone’s credentials stay encrypted and secure.

To get an object out of the cache in the previous version the method signature was:

getcacheobject(string $url, string $host, unsignedInt $port)

$url in this case is what is commonly referred to as the “path” in a standard URL; e.g. /en-US/index.html. That’s simple enough to use, let’s see what the new version has:

getcacheobject(string $url, unsignedLong $locator, string $host, unsignedInt $port [...]

$locator is an internal id for a cache object which is defintitely something I should be able to use to expire content, but what is it doing in the middle of my url+host pair? The error messages are happy to let me know that I can’t pass both $url and $locator to the same call and if I pass $url I must pass $host. That means I will always have be passing a null value in either the first or second parameter.

And if you’re thinking it’s for consistency with other methods, think again. To flush an object we’ve got:

flushcacheobject(unsignedLong $locator, string $url, string $host, unsignedInt $port [...]

Moral of the story: planning and consistency makes happy programmers; particularly with an API.

I downloaded 1.1.15 and 1.1.19, set up a couple symlinks to swap them into my dev copy and looked at how hard it would be to upgrade.

Hats off to the CakePHP developers. After merging in a short patch to the core CakePHP session code, most of the site worked right out of the box. I made a few minor tweaks to our code for things that had changed (and filed ticket 4140 for them) but all in all it was pretty painless. Nice work guys. It shows a lot of planning and foresight to make a framework that doesn’t have a bunch of code-breaking changes after years of development.

Turns out CakePHP considers a TINYINT(1) to be a Boolean. Judging from all the support tickets that have been filed, I’m not the first person to get taken off guard by this behavior. When I asked about it on IRC, the response was that since MySQL considers a TINYINT(1) to be a Boolean, CakePHP does too. That’s not true.

From the MySQL manual:

As of MySQL 5.0.3, a BIT data type is available for storing bit-field values. (Before 5.0.3, MySQL interprets BIT as TINYINT(1).)

That’s saying if I request a BIT it will make it a TINYINT, not if I request a TINYINT it will make it a BIT. Having a framework change the definitions of my database columns sounds crazy to me, but judging from the ticket filed in 2006 this has been CakePHP’s policy for a long time. Despite the long standing precedent I can’t find any documentation about it online other than the closing remarks of those support tickets.

Today’s subject is anonymous sessions. Frameworks (and developers) love them because they are simple and convenient, but it comes at a cost. Keeping track of sessions for every visitor on a high traffic site is far too expensive to be practical. Developers should know how to work around this, but their frameworks need to support them.

The first framework on my mind is Drupal. I filed an issue last year that Drupal should support disabling anonymous sessions. It’s still unassigned so I’m guessing it’s not a high priority, but it was one of the main things that made me choose not to use Drupal on mozilla.com. I wrote some ideas on how to handle it and got some responses from people suffering the same fate. No word on any progress though.

The second framework, CakePHP, has an AUTO_SESSION variable that, just like $cacheQueries, is far to easy to misplace faith in.

By setting AUTO_SESSION to false, you can’t read or write to the session. Working as advertised? Not so much. If you take a closer look at what’s actually happening you’ll see that the session is still getting started, it’s just that CakePHP is blocking your access to it. Even with AUTO_SESSION off, a cookie with a unique ID is set, and a row is still inserted into the sessions table. That last part almost brought down AMO last night. I wrote a patch that disables anonymous sessions for real, but anyone that has talked to me about patching core code knows I don’t like to do it.

When you’re writing code, framework or not, don’t forget about scalability.