It turned out pretty well so here is addons.mozilla.org development so far for 2011 (in HD!):

(Warning: prefetching is off but if you click play you’re in for an 80MB video.)

The gource docs are easy to read if you want to do this for your project, but for the record this is what I ran:

gource --viewport 1280x720 \
--user-image-dir ~/sandbox/zamboni/.git/avatar/ \
--title "addons.mozilla.org" \
--auto-skip-seconds 1 \
--seconds-per-day 1 \
--start-position .715 \
--max-file-lag 0.5 \
--max-files 5000 \
--camera-mode track \
-o -


Piped to:

ffmpeg -y \
-r 60 \
-f image2pipe \
-vcodec ppm \
-i - \
-vcodec libtheora \
-b 10000K \
~/out.ogv


That gave me a 180MB uncompressed ogv. The uncompressed version looks far nicer, but that’s a lot of bandwidth for a random video so I cut it down with ffmpeg2thoerea (anyone know the switch to do this directly in the ffmpeg command?):

ffmpeg2theora -v 4
~/out.ogv
--optimize
--noaudio
-o amo-2011-development.ogv


With the launch of the new Add-on Builder (still an alpha product, but usable) I decided to make my first Jetpack to fix this.

In only about 20 lines of code I was able to look for any 4xx or 5xx errors in the HTTP traffic and show a brief notification to the user about what went wrong and where. The builder was a great development experience (lack of documentation aside) and was a breeze to do something relatively complex.

If you’ve wanted a similar add-on, feel free to use Status Watch. It’s a Jetpack so you’ll need Firefox 4, but on the bright side, you won’t even need to restart the browser. Just click and go.

I’ve had some requests to support a whitelist of sites so you don’t get notifications all over the web and I’ll add that when I get time. It’s surprising how many sites have 403s and 404s though.

Update 2011-02-17: Version 2 is on AMO now which supports a whitelist. See the AMO page for details.

I stayed up late a few nights ago and wrote a script to raise an alert when something has changed. On its first run the script will recursively walk a directory tree hashing each file and storing the hashes in the directory (in an md5sum compatible formatted file). On subsequent runs it will begin tracking new files automatically but it will also print messages for missing and changed files. By saving the checksums in each directory it becomes portable – you can copy a directory somewhere else and still be able to verify nothing changed (a quick md5sum -c checksums.txt will let you know).

By default the script only prints messages when it sees something fishy so it’s perfect to drop into cron and it uses exit statuses so it’ll work for nagios too. I’ve been running it for a few months and have found a couple files that have changed – nothing critical yet but it’s nice to know it’s there.

My friend sent me a link to conque this morning which I’m so taken with that it’s prompted me to drop coding and write a blog post.

Conque lets me do exactly what I’ve wanted, create shells within Vim. In insert mode you interact with the shell as expected. In normal mode you can scroll back through your history, yank text into buffers, paste and manipulate text. Best thing ever.

Two files open on the left, three shells on the right; iPython, MySQL, and Django’s web server – all with syntax highlighting. ]]> http://micropipes.com/blog/2010/08/13/cli-split-windows-in-vim/feed/ 3 http://micropipes.com/blog/2010/05/03/addons-mozilla-org-%e2%99%a5s-unit-tests-again/ http://micropipes.com/blog/2010/05/03/addons-mozilla-org-%e2%99%a5s-unit-tests-again/#comments Mon, 03 May 2010 17:34:44 +0000 Wil Clouser http://micropipes.com/blog/?p=142 AMO has had an on-again off-again relationship with unit tests. A little over a year ago we had a thousand unit tests that sort of, mostly, ran. The problem is, PHP unit testing just isn’t as good as it should be. CakePHP relies on SimpleTest, one of the main PHP test suites. It worked relatively well for a small number of tests, but as our suite grew, so did our troubles.

Our main issue was hitting a memory limit or the max execution time. We hit the limits often for a variety of reasons, some legitimate bugs, and some because we tried to hack around things to make the tests run. If we change the limits we affect the tests because they are running within the same environment. There wasn’t really a concept of fixtures then, although it looks like CakePHP has stepped up there. The simple test web runner was hard to use and the mock objects were sometimes a little too mocked and missing some attributes.

All in all it was a heroic effort to get that many tests, but we didn’t maintain it because they were so slow to write and difficult to run. Testing can be a pain to write, sure, but it shouldn’t be a burden like that. Enter Django’s testing suite (built on top of Python’s unittest). It has most of our complaints handled out of the box. It’s very well documented, considers a lot of aspects of testing, supports fixtures, a built-in client, etc. It’s a well thought out framework to build tests on.

We’re being more vigilant about requiring tests this time around, but they also aren’t as frustrating to write. When you write them they actually work and they stay working. Most of what you want is built in already. For example, I wrote the password reset form we needed on AMO in Django. With CakePHP and SimpleTest I’d have no idea how to test that the email was actually working. It’s apparently possible with a SimpleTest add-on and enough code that I have to scroll in my browser. With Django’s test suite the actual code was 5 lines, 3 of which were assertions:

    def test_request_success(self):
        self.client.post('/en-US/firefox/users/pwreset',
                        {'email': self.user.email})

        eq_(len(mail.outbox), 1)
        assert mail.outbox[0].subject.find('Password reset') == 0
        assert mail.outbox[0].body.find('pwreset/%s' % self.uidb36) > 0

With the power of the new test suite we’re once again writing and maintaining our unit tests – currently at around 390 tests and increasing steadily. Plenty of people have written about why unit tests are important so I won’t belabor the point, but I will mention that it’s a great feeling to be able to commit something and be confident it hasn’t affected other parts of the site. It’s almost as good of a feeling when you write your code and a completely different test fails pointing out a case that you didn’t even consider but one that would soak up developer time trying to debug down the road.

Building on a foundation that takes testing seriously is great.

AMO is the first Mozilla Webdev site to use continuous integration, and it’s been a long time coming. With the way it’s currently configured we’ve got code coverage trending, unit test trending, code quality trending, as well as detailed reports for all the above for every single check in.

If anything fails or oversteps a threshold our IRC bot complains and we can get it fixed up quickly. It’s a boon to productivity to know that all the code being checked in is being tested automatically, plus it gives everyone a stable state to compare to.

Thanks to everyone that helped get Hudson going, from the people that write it, to the IT team that keeps it alive, to the webdev team that helped work out the kinks.

We hit some limitations with the NetScaler that we weren’t happy with. Cost aside, it ignored some pretty standard stuff like the HTTP Vary Header. After working around that for years we switched to the horizontally scalable Zeus Traffic Manager (at that time, referred to as ZXTM). We’ve been pleased with our choice and six months ago I wrote a similar PHP library that allows you to connect to Zeus’s API. Time and priorities being what they are, we never implemented it in production.

Finally, the real point to this post, last night I wrote a python library that will expire content from Zeus. We’ll roll this into our migration and waiting on content to expire from Zeus should be a thing of the past.

As always, if you can use the libraries, feel free. They all have READMEs with examples.

First out of the chute was the .po files. It turns out the actual formatting is different between the two languages. PHP uses %1$s for its substitutions, but python uses either named variables like (num)s or integers like {0}. For the record, they both support %s when you don’t need to order the substitutions.
PHP example:
I have %2$s apples and %1$s oranges
Python example:
I have {1} apples and {0} oranges

Since I’ve worked with the Translate Toolkit before, I decided to write a script to convert between the two formats. If you find yourself in the same unfortunate boat as me, behold
phppo2pypo and pypo2phppo to convert between the two types.

Crisis averted, right? Oh, that’s just scratching the surface. Remember how happy I was that PHP finally started supporting msgctxt? Well, Python has had a patch for it since 2008 but no one has bothered to land it. I wrote a new ugettext() and ungettext() that recognizes context in the .po files. To use simply do: from l10n import ugettext as _ at the top of your file.

Along with adding msgctxt support, those two functions also collapse consecutive white space. We’re using Jinja2 with Babel and the i18n extension as our template engine. Jinja2 has a concept of stripping white space from the beginning or end of a string but does nothing about the middle. A paragraph of text in a Jinja2 template would look like:

{% trans -%}Mozilla is providing links to these applications
as a courtesy, and makes no representations regarding the
applications or any information related thereto. Any questions,
complaints or claims regarding the applications must be
directed to the appropriate software vendor.
{%- endtrans %}


That’s a decent looking template, right? Yeah, well, when Babel extracts that, it includes all the line breaks too, giving you something like this. The localizers would revolt if I sent them that, so I added in auto white-space collapsing. Getting Babel to use the new functions means a new extraction script.

At this point, we’re extracting strings from our new code and we can convert between Python and PHP files. All we need now is a Frankenstein mix of xgettext functions to act as glue. Meet the amalgamate script that uses the pypo2php scripts, concatenates the .pot files, and merge updates each locales .po file. After that it’s quick tweaks to the build scripts to create z-messages.po files and we’re done.

So, all that said, the new process for L10n, while we’re in this transitional phase, is:

1. From the PHP code, run locale/extract-po-remora.sh. That pulls everything from all the PHP files, creates locale/r-keys.pot, updates the messages.po file for each locale, and compiles them. Life used to be so simple.
2. From the python code, make sure you’re up to date, then run ./manage.py extract. That will pull everything from the python code and templates and create locale/z-keys.pot.
3. Run ./manage.py amalgamate. That will merge the z-keys.pot into the PHP messages.po files.
4. Localizers can make their changes as usual, and commit back to messages.po.
5. From PHP, locale/copy-to-zamboni.py locale will create z-messages.po files in the Python format. We could skip right to .mo files, but in case something goes wrong I want to see the .po files.
6. Then, like today, locale/compile-mo.sh locale will compile all the .po files.

After all those steps are done, we’ve got duplicate .mo files, aside from formatting, and each application can look at its own .mo to get the strings it needs. All this code is just a big band-aid and there are plenty of things that are more fun than juggling L10n between two applications across two RCSs. But we knew what we were getting in to. I’ll post something more positive later to help justify it.

I can look at them any time, and there are a few on flickr that other people can look at, but what about people that aren’t tech savvy or are really busy and get caught up in the daily grind?

I was thinking, the answer can be a really simple script running from cron, say, weekly. It picks a random photo from a directory and emails it to a group of people. That’s it. The idea is:

• it’s low tech compared to RSS feeds and social networking sites. This “just works” with the tools people (potentially, low-tech people) are already used to using.
• in a similar vein, commenting and discussion are built in if you feel like it.
• it’s focused on the people on the CC list. Sending out an old photo that is relevant just to those people has a lot more effect than one scrolling by on flickr. It’s got a personal touch.
• it’s automatic. You wake up Monday morning, drag yourself to work, and there is a photo from 15 years ago in your inbox and you can laugh about how bad your hair was. Awesome.

So, I thought I would try it and cooked up audreytoo. Basically, you seed a directory with copies of the pictures you want to send out, add the script to a cron job, and it does it’s thing. There are a couple other features in the script that you could look at if you want to get fancier.

I realize there is some irony in using an automated script to say “I’m thinking of you” but as long as I’m on the CC list it’s still true. Anyway, I’m sharing the code in case anyone else can find value in it.

On the SVN server the first thing we did was to create a special Verbatim user that can commit to SVN via SSH using a generated key. We copied this key to the Verbatim host which allowed us to commit as the verbatim user without typing a username or password.

The only thing that was added to the Verbatim code was a patch that Dan Schafer cooked up that sets an SVN revision property, translate:author, to the name of the current user. When the user clicks “commit” this property is set and sent along with the commit.

At this point we could commit from the application but it still goes to the application as the Verbatim user. We used SVN’s hooks to take the next step.

The first script we changed was the pre-revprop-change hook. This controls what special revision properties a user can modify when they commit. Our script adds the ability to modify svn:author and translate:author. Before allowing the modifications the script checks if the user committing is the special verbatim user to prevent anyone from committing as someone else.

Next we added a post-commit script that looks for the translate:author property. If it’s found it will take that value, replace svn:author, and remove translate:author; effectively making whatever was in translate:author the real author. This is a non-versioned change which means there is no commit that needs to happen – the new author is set immediately.

With these scripts in place we can commit as anyone from the application and everyone’s credentials stay encrypted and secure.