My friend sent me a link to conque this morning which I’m so taken with that it’s prompted me to drop coding and write a blog post.

Conque lets me do exactly what I’ve wanted, create shells within Vim. In insert mode you interact with the shell as expected. In normal mode you can scroll back through your history, yank text into buffers, paste and manipulate text. Best thing ever.

Two files open on the left, three shells on the right; iPython, MySQL, and Django’s web server – all with syntax highlighting. ]]> http://micropipes.com/blog/2010/08/13/cli-split-windows-in-vim/feed/ 3 http://micropipes.com/blog/2010/05/03/addons-mozilla-org-%e2%99%a5s-unit-tests-again/ http://micropipes.com/blog/2010/05/03/addons-mozilla-org-%e2%99%a5s-unit-tests-again/#comments Mon, 03 May 2010 17:34:44 +0000 Wil Clouser http://micropipes.com/blog/?p=142 AMO has had an on-again off-again relationship with unit tests. A little over a year ago we had a thousand unit tests that sort of, mostly, ran. The problem is, PHP unit testing just isn’t as good as it should be. CakePHP relies on SimpleTest, one of the main PHP test suites. It worked relatively well for a small number of tests, but as our suite grew, so did our troubles.

Our main issue was hitting a memory limit or the max execution time. We hit the limits often for a variety of reasons, some legitimate bugs, and some because we tried to hack around things to make the tests run. If we change the limits we affect the tests because they are running within the same environment. There wasn’t really a concept of fixtures then, although it looks like CakePHP has stepped up there. The simple test web runner was hard to use and the mock objects were sometimes a little too mocked and missing some attributes.

All in all it was a heroic effort to get that many tests, but we didn’t maintain it because they were so slow to write and difficult to run. Testing can be a pain to write, sure, but it shouldn’t be a burden like that. Enter Django’s testing suite (built on top of Python’s unittest). It has most of our complaints handled out of the box. It’s very well documented, considers a lot of aspects of testing, supports fixtures, a built-in client, etc. It’s a well thought out framework to build tests on.

We’re being more vigilant about requiring tests this time around, but they also aren’t as frustrating to write. When you write them they actually work and they stay working. Most of what you want is built in already. For example, I wrote the password reset form we needed on AMO in Django. With CakePHP and SimpleTest I’d have no idea how to test that the email was actually working. It’s apparently possible with a SimpleTest add-on and enough code that I have to scroll in my browser. With Django’s test suite the actual code was 5 lines, 3 of which were assertions:

    def test_request_success(self):
        self.client.post('/en-US/firefox/users/pwreset',
                        {'email': self.user.email})

        eq_(len(mail.outbox), 1)
        assert mail.outbox[0].subject.find('Password reset') == 0
        assert mail.outbox[0].body.find('pwreset/%s' % self.uidb36) > 0

With the power of the new test suite we’re once again writing and maintaining our unit tests – currently at around 390 tests and increasing steadily. Plenty of people have written about why unit tests are important so I won’t belabor the point, but I will mention that it’s a great feeling to be able to commit something and be confident it hasn’t affected other parts of the site. It’s almost as good of a feeling when you write your code and a completely different test fails pointing out a case that you didn’t even consider but one that would soak up developer time trying to debug down the road.

Building on a foundation that takes testing seriously is great.

We hit some limitations with the NetScaler that we weren’t happy with. Cost aside, it ignored some pretty standard stuff like the HTTP Vary Header. After working around that for years we switched to the horizontally scalable Zeus Traffic Manager (at that time, referred to as ZXTM). We’ve been pleased with our choice and six months ago I wrote a similar PHP library that allows you to connect to Zeus’s API. Time and priorities being what they are, we never implemented it in production.

Finally, the real point to this post, last night I wrote a python library that will expire content from Zeus. We’ll roll this into our migration and waiting on content to expire from Zeus should be a thing of the past.

As always, if you can use the libraries, feel free. They all have READMEs with examples.

First out of the chute was the .po files. It turns out the actual formatting is different between the two languages. PHP uses %1$s for its substitutions, but python uses either named variables like (num)s or integers like {0}. For the record, they both support %s when you don’t need to order the substitutions.
PHP example:
I have %2$s apples and %1$s oranges
Python example:
I have {1} apples and {0} oranges

Since I’ve worked with the Translate Toolkit before, I decided to write a script to convert between the two formats. If you find yourself in the same unfortunate boat as me, behold
phppo2pypo and pypo2phppo to convert between the two types.

Crisis averted, right? Oh, that’s just scratching the surface. Remember how happy I was that PHP finally started supporting msgctxt? Well, Python has had a patch for it since 2008 but no one has bothered to land it. I wrote a new ugettext() and ungettext() that recognizes context in the .po files. To use simply do: from l10n import ugettext as _ at the top of your file.

Along with adding msgctxt support, those two functions also collapse consecutive white space. We’re using Jinja2 with Babel and the i18n extension as our template engine. Jinja2 has a concept of stripping white space from the beginning or end of a string but does nothing about the middle. A paragraph of text in a Jinja2 template would look like:

{% trans -%}Mozilla is providing links to these applications
as a courtesy, and makes no representations regarding the
applications or any information related thereto. Any questions,
complaints or claims regarding the applications must be
directed to the appropriate software vendor.
{%- endtrans %}


That’s a decent looking template, right? Yeah, well, when Babel extracts that, it includes all the line breaks too, giving you something like this. The localizers would revolt if I sent them that, so I added in auto white-space collapsing. Getting Babel to use the new functions means a new extraction script.

At this point, we’re extracting strings from our new code and we can convert between Python and PHP files. All we need now is a Frankenstein mix of xgettext functions to act as glue. Meet the amalgamate script that uses the pypo2php scripts, concatenates the .pot files, and merge updates each locales .po file. After that it’s quick tweaks to the build scripts to create z-messages.po files and we’re done.

So, all that said, the new process for L10n, while we’re in this transitional phase, is:

1. From the PHP code, run locale/extract-po-remora.sh. That pulls everything from all the PHP files, creates locale/r-keys.pot, updates the messages.po file for each locale, and compiles them. Life used to be so simple.
2. From the python code, make sure you’re up to date, then run ./manage.py extract. That will pull everything from the python code and templates and create locale/z-keys.pot.
3. Run ./manage.py amalgamate. That will merge the z-keys.pot into the PHP messages.po files.
4. Localizers can make their changes as usual, and commit back to messages.po.
5. From PHP, locale/copy-to-zamboni.py locale will create z-messages.po files in the Python format. We could skip right to .mo files, but in case something goes wrong I want to see the .po files.
6. Then, like today, locale/compile-mo.sh locale will compile all the .po files.

After all those steps are done, we’ve got duplicate .mo files, aside from formatting, and each application can look at its own .mo to get the strings it needs. All this code is just a big band-aid and there are plenty of things that are more fun than juggling L10n between two applications across two RCSs. But we knew what we were getting in to. I’ll post something more positive later to help justify it.

I can look at them any time, and there are a few on flickr that other people can look at, but what about people that aren’t tech savvy or are really busy and get caught up in the daily grind?

I was thinking, the answer can be a really simple script running from cron, say, weekly. It picks a random photo from a directory and emails it to a group of people. That’s it. The idea is:

• it’s low tech compared to RSS feeds and social networking sites. This “just works” with the tools people (potentially, low-tech people) are already used to using.
• in a similar vein, commenting and discussion are built in if you feel like it.
• it’s focused on the people on the CC list. Sending out an old photo that is relevant just to those people has a lot more effect than one scrolling by on flickr. It’s got a personal touch.
• it’s automatic. You wake up Monday morning, drag yourself to work, and there is a photo from 15 years ago in your inbox and you can laugh about how bad your hair was. Awesome.

So, I thought I would try it and cooked up audreytoo. Basically, you seed a directory with copies of the pictures you want to send out, add the script to a cron job, and it does it’s thing. There are a couple other features in the script that you could look at if you want to get fancier.

I realize there is some irony in using an automated script to say “I’m thinking of you” but as long as I’m on the CC list it’s still true. Anyway, I’m sharing the code in case anyone else can find value in it.

To get an object out of the cache in the previous version the method signature was:

getcacheobject(string $url, string $host, unsignedInt $port)

$url in this case is what is commonly referred to as the “path” in a standard URL; e.g. /en-US/index.html. That’s simple enough to use, let’s see what the new version has:

getcacheobject(string $url, unsignedLong $locator, string $host, unsignedInt $port [...]

$locator is an internal id for a cache object which is defintitely something I should be able to use to expire content, but what is it doing in the middle of my url+host pair? The error messages are happy to let me know that I can’t pass both $url and $locator to the same call and if I pass $url I must pass $host. That means I will always have be passing a null value in either the first or second parameter.

And if you’re thinking it’s for consistency with other methods, think again. To flush an object we’ve got:

flushcacheobject(unsignedLong $locator, string $url, string $host, unsignedInt $port [...]

Moral of the story: planning and consistency makes happy programmers; particularly with an API.

Turns out CakePHP considers a TINYINT(1) to be a Boolean. Judging from all the support tickets that have been filed, I’m not the first person to get taken off guard by this behavior. When I asked about it on IRC, the response was that since MySQL considers a TINYINT(1) to be a Boolean, CakePHP does too. That’s not true.

From the MySQL manual:

As of MySQL 5.0.3, a BIT data type is available for storing bit-field values. (Before 5.0.3, MySQL interprets BIT as TINYINT(1).)

That’s saying if I request a BIT it will make it a TINYINT, not if I request a TINYINT it will make it a BIT. Having a framework change the definitions of my database columns sounds crazy to me, but judging from the ticket filed in 2006 this has been CakePHP’s policy for a long time. Despite the long standing precedent I can’t find any documentation about it online other than the closing remarks of those support tickets.

Today’s subject is anonymous sessions. Frameworks (and developers) love them because they are simple and convenient, but it comes at a cost. Keeping track of sessions for every visitor on a high traffic site is far too expensive to be practical. Developers should know how to work around this, but their frameworks need to support them.

The first framework on my mind is Drupal. I filed an issue last year that Drupal should support disabling anonymous sessions. It’s still unassigned so I’m guessing it’s not a high priority, but it was one of the main things that made me choose not to use Drupal on mozilla.com. I wrote some ideas on how to handle it and got some responses from people suffering the same fate. No word on any progress though.

The second framework, CakePHP, has an AUTO_SESSION variable that, just like $cacheQueries, is far to easy to misplace faith in.

By setting AUTO_SESSION to false, you can’t read or write to the session. Working as advertised? Not so much. If you take a closer look at what’s actually happening you’ll see that the session is still getting started, it’s just that CakePHP is blocking your access to it. Even with AUTO_SESSION off, a cookie with a unique ID is set, and a row is still inserted into the sessions table. That last part almost brought down AMO last night. I wrote a patch that disables anonymous sessions for real, but anyone that has talked to me about patching core code knows I don’t like to do it.

When you’re writing code, framework or not, don’t forget about scalability.

One evening I decided I wanted to try making an IRC bot (because you know we need more of those!). I’d only worked with Python briefly before, but armed with the confidence only a language that you don’t know the shortcomings of can bring, I started writing code. After less than an hour and about 85 well-spaced lines of code later, Aaplsauce was born. All it does is sit idle in a channel and watch for someone to say something about AAPL. After someone mentions the magic phrase the bot looks up the price and prints it in the channel. How convenient!

So, nearly worthless? Absolutely. But the code is short and easy to write and python made that possible. Thx Python.

(yeah, if you read the code there is one other command the bot understands and, no, there’s no auth on it. Don’t make me do it! )

The add-on is available, but it’s currently in the sandbox (which means you’ll have to login if you want to see it). If you already have an account, feel free to add a review so I can get started on the process to get onto the public side of AMO.

Once the add-on hits the public side, anyone with an older version will get automatically updated which should help anyone experiencing the “disappearing threads” from version 0.3.