AMO has had an on-again off-again relationship with unit tests. A little over a year ago we had a thousand unit tests that sort of, mostly, ran. The problem is, PHP unit testing just isn’t as good as it should be. CakePHP relies on SimpleTest, one of the main PHP test suites. It worked relatively well for a small number of tests, but as our suite grew, so did our troubles.
And we love it. :)
It's time to hail another milestone for AMO in our epic push for improvements in 2010. This time I'm happy to announce our Hudson continuous integration server which has been humming along for a few months. Hudson Integration Screenshot. Click to enlarge. AMO is the first Mozilla Webdev site to use continuous integration, and it's been a long time coming. With the way it's currently configured we've got code coverage trending, unit test trending, code quality trending, as well as detailed reports for all the above for every single check in. If anything fails or oversteps a threshold our IRC bot complains and we can get it fixed up quickly. It's a boon to productivity to know that all the code being checked in is being tested automatically, plus it gives everyone a stable state to compare to. Thanks to everyone that helped get Hudson going, from the people that write it, to the IT team that keeps it alive, to the webdev team that helped work out the kinks.
The first front end cache we used on AMO was the Citrix NetScaler. I've complained about it's API before but apparently never announced the library I wrote to purge items from the cache. So, a little late, but I have some reusable PHP code that will talk to your NetScaler and let you expire objects. We hit some limitations with the NetScaler that we weren't happy with. Cost aside, it ignored some pretty standard stuff like the HTTP Vary Header. After working around that for years we switched to the horizontally scalable Zeus Traffic Manager (at that time, referred to as ZXTM). We've been pleased with our choice and six months ago I wrote a similar PHP library that allows you to connect to Zeus's API. Time and priorities being what they are, we never implemented it in production. Finally, the real point to this post, last night I wrote a python library that will expire content from Zeus. We'll roll this into our migration and waiting on content to expire from Zeus should be a thing of the past. As always, if you can use the libraries, feel free. They all have READMEs with examples.
I reached my hand into the barrel of problems our migration to Python is going to cause and came up with Localization. It figures. First out of the chute was the .po files. It turns out the actual formatting is different between the two languages. PHP uses %1$s for its substitutions, but python uses either named variables like (num)s or integers like {0}. For the record, they both support %s when you don't need to order the substitutions. PHP example: I have %2$s apples and %1$s oranges Python example: I have {1} apples and {0} oranges Since I've worked with the Translate Toolkit before, I decided to write a script to convert between the two formats. If you find yourself in the same unfortunate boat as me, behold phppo2pypo and pypo2phppo to convert between the two types. Crisis averted, right? Oh, that's just scratching the surface. Remember how happy I was that PHP finally started supporting msgctxt? Well, Python has had a patch for it since 2008 but no one has bothered to land it. I wrote a new ugettext() and ungettext() that recognizes context in the .po files. To use simply do: from l10n import ugettext as _ at the top of your file. Along with adding msgctxt support, those two functions also collapse consecutive white space. We're using Jinja2 with Babel and the i18n extension as our template engine. Jinja2 has a concept of stripping white space from the beginning or end of a string but does nothing about the middle. A paragraph of text in a Jinja2 template would look like: {% trans -%}Mozilla is providing links to these applications as a courtesy, and makes no representations regarding the applications or any information related thereto. Any questions, complaints or claims regarding the applications must be directed to the appropriate software vendor. {%- endtrans %} That's a decent looking template, right? Yeah, well, when Babel extracts that, it includes all the line breaks too, giving you something like this. The localizers would revolt if I sent them that, so I added in auto white-space collapsing. Getting Babel to use the new functions means a new extraction script. At this point, we're extracting strings from our new code and we can convert between Python and PHP files. All we need now is a Frankenstein mix of xgettext functions to act as glue. Meet the amalgamate script that uses the pypo2php scripts, concatenates the .pot files, and merge updates each locales .po file. After that it's quick tweaks to the build scripts to create z-messages.po files and we're done. So, all that said, the new process for L10n, while we're in this transitional phase, is: From the PHP code, run locale/extract-po-remora.sh. That pulls everything from all the PHP files, creates locale/r-keys.pot, updates the messages.po file for each locale, and compiles them. Life used to be so simple. From the python code, make sure you're up to date, then run ./manage.py extract. That will pull everything from the python code and templates and create locale/z-keys.pot. Run ./manage.py amalgamate. That will merge the z-keys.pot into the PHP messages.po files. Localizers can make their changes as usual, and commit back to messages.po. From PHP, locale/copy-to-zamboni.py locale will create z-messages.po files in the Python format. We could skip right to .mo files, but in case something goes wrong I want to see the .po files. Then, like today, locale/compile-mo.sh locale will compile all the .po files. After all those steps are done, we've got duplicate .mo files, aside from formatting, and each application can look at its own .mo to get the strings it needs. All this code is just a big band-aid and there are plenty of things that are more fun than juggling L10n between two applications across two RCSs. But we knew what we were getting in to. I'll post something more positive later to help justify it. :)