First out of the chute was the .po files. It turns out the actual formatting is different between the two languages. PHP uses %1$s for its substitutions, but python uses either named variables like (num)s or integers like {0}. For the record, they both support %s when you don’t need to order the substitutions.
PHP example:
I have %2$s apples and %1$s oranges
Python example:
I have {1} apples and {0} oranges

Since I’ve worked with the Translate Toolkit before, I decided to write a script to convert between the two formats. If you find yourself in the same unfortunate boat as me, behold
phppo2pypo and pypo2phppo to convert between the two types.

Crisis averted, right? Oh, that’s just scratching the surface. Remember how happy I was that PHP finally started supporting msgctxt? Well, Python has had a patch for it since 2008 but no one has bothered to land it. I wrote a new ugettext() and ungettext() that recognizes context in the .po files. To use simply do: from l10n import ugettext as _ at the top of your file.

Along with adding msgctxt support, those two functions also collapse consecutive white space. We’re using Jinja2 with Babel and the i18n extension as our template engine. Jinja2 has a concept of stripping white space from the beginning or end of a string but does nothing about the middle. A paragraph of text in a Jinja2 template would look like:

{% trans -%}Mozilla is providing links to these applications
as a courtesy, and makes no representations regarding the
applications or any information related thereto. Any questions,
complaints or claims regarding the applications must be
directed to the appropriate software vendor.
{%- endtrans %}


That’s a decent looking template, right? Yeah, well, when Babel extracts that, it includes all the line breaks too, giving you something like this. The localizers would revolt if I sent them that, so I added in auto white-space collapsing. Getting Babel to use the new functions means a new extraction script.

At this point, we’re extracting strings from our new code and we can convert between Python and PHP files. All we need now is a Frankenstein mix of xgettext functions to act as glue. Meet the amalgamate script that uses the pypo2php scripts, concatenates the .pot files, and merge updates each locales .po file. After that it’s quick tweaks to the build scripts to create z-messages.po files and we’re done.

So, all that said, the new process for L10n, while we’re in this transitional phase, is:

1. From the PHP code, run locale/extract-po-remora.sh. That pulls everything from all the PHP files, creates locale/r-keys.pot, updates the messages.po file for each locale, and compiles them. Life used to be so simple.
2. From the python code, make sure you’re up to date, then run ./manage.py extract. That will pull everything from the python code and templates and create locale/z-keys.pot.
3. Run ./manage.py amalgamate. That will merge the z-keys.pot into the PHP messages.po files.
4. Localizers can make their changes as usual, and commit back to messages.po.
5. From PHP, locale/copy-to-zamboni.py locale will create z-messages.po files in the Python format. We could skip right to .mo files, but in case something goes wrong I want to see the .po files.
6. Then, like today, locale/compile-mo.sh locale will compile all the .po files.

After all those steps are done, we’ve got duplicate .mo files, aside from formatting, and each application can look at its own .mo to get the strings it needs. All this code is just a big band-aid and there are plenty of things that are more fun than juggling L10n between two applications across two RCSs. But we knew what we were getting in to. I’ll post something more positive later to help justify it.

Migrating from CakePHP to Django

This is a big undertaking and we’ve been discussing it for quite a while. We’re currently the highest trafficked site on the internet using CakePHP and along with that we’ve run into a lot of frustrating issues. CakePHP has serviced AMO well for several years, so it’s not my intention to bad mouth it here, but I do want to give a fair summary of why we’re moving on. Please also note that AMO is still running on CakePHP 1.1 which is, I think, a year out of date? Three substantial issues:

• Useful Database Abstraction Layer: CakePHP has a concept of database abstraction, but we didn’t find it powerful enough. When it did work it would return enormous nested arrays of data causing massive CPU and memory usage (out of memory errors plague us on AMO). When it didn’t work, we’d end up doing queries directly which kind of defeats the purpose. We couldn’t use prepared statements so we’d have to escape variables ourselves. There was no effective caching built-in and since we just had huge arrays as a response there was no effective way to invalidate the cache we were using (see: Caching is easy; Expiration is hard). The DB layer should return objects that are easy to cache and easy to invalidate. The built-in Django database classes (combined with memcache) should work fine for us here.
• Effective unit tests: I’ve beat the drum about our unit tests before but the simple matter is that it’s really difficult to do them right with the tools we are using. Our test data is already very limited, but if we try to run all our tests right now they’ll run out of memory (and take forever). The CakePHP method of mocking controllers and models was inadequate for what we needed and difficult to deal with. We want our unit tests to run quickly, from the command line, and be independent from each other so there aren’t intermittent problems to waste our time with. We’ll be using Django’s built-in testing framework.
• Better debugging: Debugging in CakePHP amounts to defining a DEBUG level and seeing what is printed on the screen (usually the giant arrays). We supplemented this with Xdebug where we needed it, but that’s still not enough. A framework should have excellent logging and on-the-fly debugging that displays a full traceback (often something will fail deep within CakePHP and we’ll get the file/line where PHP gave up, but not the line in our code that started the problem), the values of variables, the page headers, server settings, SQL that was run, what views and elements are in use, etc. We’re planning on using a combination of pdb, IPython, and the django-debug-toolbar to make all of this easily accessible while developing.

Those are the major issues we’re having right now, but if you want to dig into the comparison some more check out our discussion wiki pages, but realize the majority of discussion happened in person.

Moving away from SVN

We moved AMO into SVN in 2006 and it’s treated us relatively well. Somewhere along the line, we decided to tag our production versions at a revision of trunk instead of keeping a separate tag and merging changes into it. It’s worked for us but it’s a hard cutoff on code changes, which means that while we’re in a code freeze no one can check anything in to trunk. As we begin to branch for larger projects this will become more of a hassle, so I’m planning on going back to a system where a production tag is created and changes are merged into it as they are ready to go live.

Most of the development team has been using git-svn for several months and, aside from the commands being far more verbose, we haven’t had many complaints. We’ve discovered Git is a much more powerful development tool and we expect to use it directly starting some time next year. As of now, we expect to maintain the /locales/ directory in SVN so this change doesn’t affect localizers but we’ll keep people notified if there are any changes to that process.

Continuous Integration

I mentioned excellent testing being one of the reasons we’re moving to Django. Along with that testing is the opportunity for continuous integration. We plan on using Hudson as the framework for our continuous integration. With excellent test coverage and quick feedback from Hudson this should drastically lower our regressions and boost our confidence when we deploy. Speaking of which…

Faster Deployment

For most of 2009 we’ve pushed on 3 week cycles. 2 weeks of development, 1 week of QA and L10n. Delays and regressions being what they are, I think we averaged a little better than a push a month. This is a fairly rapid cycle for a lot of development shops, but I feel like it’s holding us back. We’ve heard a lot of success stories about shorter cycles and I’d like to aim for deployment (optionally, of course) of a few times per week. By shortening the development cycle we reduce the stress of:

• the developers: Everyone likes to see what they’ve done go out quicker and it means less conflicts with others when the patches are smaller.
• the QA team: Right now we dump 2 weeks of work on them and say we need it done right away. With smaller cycles they can verify small changes as they go and not be overwhelmed.
• the infrastructure team: Smaller changes means less to go wrong and with a continuous integration server and some automation they can have minimal involvement with the whole process.
• the localizers: Every time we release we dump a bunch of changes on these fantastic people and tell them we need them back in a week. Most of the time they plow forward and get them done on time. If they don’t though, they are stuck with waiting for the next 3 week cycle. If we push often, it’s not a big deal.
• the product managers: These guys come up with crazy ideas for us to implement and then they stare at graphs and numbers to see if it worked. With shorter cycles they can get faster feedback about what works and what doesn’t.
• the users: Faster release cycles means bugs that are fixed in the repository are fixed on the live site sooner. ’nuff said.

Process Data Offline

Much of AMO relies on cron jobs to get things done. All the statistics, add-on download numbers, how popular an add-on is, all the star rating calculations, any cleanup or maintenance tasks – these are all run via cron and they are so intensive that the database has trouble keeping up. We’re planning on utilizing Gearman to farm all this work out to other machines in incremental pieces instead of single huge queries. Any heavy calculating that can be done offline will be moved to these external processors which should help improve the speed of the site and make all our statistics more reliable (as currently the cron jobs have a tendency to fail before they are complete).

Improve the Documentation

Documentation is a noble goal of many developers but it rarely gets enough attention. We evaluated our current documentation and found it is woefully out of date. By being on a wiki that is rarely used it doesn’t get updated except when someone tries to use it and sees it’s not right. We’re hoping to change that by moving the developer documentation into the code repository itself. We’ll be able to integrate with generated API docs, style the docs however we want, and check in changes right along with our code patches. When someone checks out a copy of AMO, they’ll get all the documentation right along with it. We’ll use Sphinx to build the docs.

The outline above details several large, high-level changes but there are a lot of other plans for smaller improvements as well. This post got a lot longer than I was expecting, but I’m really excited about the direction AMO is headed for 2010. As these changes are implemented the site will become more responsive and reliable, and we’ll be able to adapt to the needs of Mozilla’s users even faster. As always, feedback and discussion are welcome and stay tuned for further back end improvements.

The framework was written in an extensible way so in addition to tweaking the built-in searches, we could also leverage external scripts. The first such script that is making it to the live site is Adrian Kalla’s localization completeness check. This script attempts to parse and record all the English string files as a baseline. Then it looks at each locale and reports any missing files, missing translations, or untranslated strings (translations that exist in the locale but are the same as English).

If you validate an extension now and only have partial L10n coverage, scroll down to the new L10n section and you should see something like this:

Thanks to RJ and Adrian for doing all the work on this.

Since that time msgctxt has become a standard part of gettext and makes my substitution string recommendation obsolete. I wanted to officially come out and say: substitution strings are a pain. The scripts we used made it manageable but finding strings in the code meant searching through the .po and not only was this painful for our developers but I think it confused contributors as well.

In our latest release, we’ve converted AMO to use regular .po files now. On the off chance someone followed my advice and would like to convert their site to regular .po files as well, Zbigniew Braniecki wrote a bit about the process and you can grab his scripts (and read about the troubles I had) at bug 501988.

Since I tagged this post hindsight I guess I should look back and conclude something too. Would I do it again? At the time, there were no great alternatives. So, yeah, I would get more opinions on whether the Gnome/KDE method was better, but I would do it again. I think it was the best choice out of several poor ones, but that doesn’t mean I’m not very happy to be rid of them.

[1] To be fair, there was a more common way, used by some KDE and Gnome projects, which was to put a delimiter in the msgid and keep the context on one side and the original string on the other. This is also pretty hacky and you can read about Gnome’s migration away from that method if you’re curious.

Allowing full Unicode would let anyone use meaningful tags in their own character sets but would prevent us from offering similar matches and common misspellings. On the other hand, we support several languages on AMO that don’t use the Latin alphabet. It stands to reason that users would search for tags in their own character sets and would get no results. There are pros and cons for each choice but we’re essentially debating the value of normalization in tagging.

After distilling all the feedback and talking amongst ourselves our overall feeling was that forcing people to convert their input into the Latin alphabet wasn’t in the users’ best interest. The Mozilla Manifesto talks about a global internet that fosters creativity and free expression. Not supporting a user’s native language when we have the option to doesn’t feel like the right path to take.

With that in mind our current plan is as follows:

• Allow full Unicode in tags to the extent we do everywhere else on AMO.
• Do no automatic character normalization. The option of manual normalization (essentially, marking some tags as equivalent) is left open as a future enhancement.
• Do automatic white space and capitalization normalization. Spaces are displayed on an add-on’s page but when searching or entering into a URL spaces are unnecessary. For example, newyork, new york and New YORk are all equivalent.
• A list of suggestions will be provided as the user types. We may attempt some simplistic character normalization in the suggestions if we can come up with a way that provides enough value to continue to use (perhaps something that is per-language).
• White space is trimmed from the beginning and end of tags before they are saved into the database.
• Tags are limited to 128 characters and add-ons are limited to 80 tags.
• Tags will be comma delimited. To include a comma in your tag you must use quotation marks. Quotation marks, whether they are matched or not, are discarded. Example: “Portland, OR” will become Portland, OR whereas Portland”, OR will become Portland and OR.

Additional feedback, as always, is welcome.

A tag library exists for CakePHP but it, and many others, are too simplistic for what we want.

We’ve written our tagging goals into a plan but have some technical details we still need to figure out. While reviewing what we have a couple questions arose that we thought people would have opinions on.

1) What should the range of allowed characters be? Our first instinct was simplicity, something like /[A-Za-z0-9-]/ (that is, all English letters and numbers and a dash). This is easy to handle on our end but leaves out everyone that doesn’t want to add tags using the English alphabet. There is some debate how useful it would be to allow other Unicode characters, particularly when you think about #2 below.

2) Tags are most useful when they are normalized. By allowing Unicode characters we run the risk of diluting our tag cloud. For example, resume and résumé are close enough that for our purposes they are equivalent. If we allow Unicode we’ll have to deal with converting characters like é to e and vice versa for searches. At that point we’ll need a list of “equivalent” characters – not impossible but it will slow things down (both development and speed of a search). The second question is: Assuming you think we should allow Unicode characters, what characters are equivalents? Here is a quick idea from php.net’s strtr() documentation:

$a = 'ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûýýþÿŔŕ';
$b = 'aaaaaaaceeeeiiiidnoooooouuuuybsaaaaaaaceeeeiiiidnoooooouuuyybyRr';

Some other aspects of our current plan are:

• Tags are not localizable in the same way as other strings on the site (like categories). There isn’t anything stopping someone from using “WebDev” as a tag or creating a new tag with “WebDev” translated in their language. However, there won’t be any relationship between the two translated tags.
• Tags are separated by spaces. Spaces within tags are allowed with quotes.
• Spaces will be preserved when displaying a tag on the add-on’s page, however, they will be removed for displaying the tag in a URL and for doing logical operations on the back end like searching. This means searching for “Portland OR” will actually be collapsed to “PortlandOR” and will match either “Portland OR” or “PortlandOR” tags. This is consistent with flickr.
• If unicode is allowed we’ll preserve characters as they are entered even if we are actually searching on their “equivalents.”

While writing code it became apparent that the framework Pootle was built on, jToolkit, had some shortcomings that were making it difficult to work with (not to mention development had been stopped on it since 2006). The decision was made to migrate the back end of Pootle from jToolkit to Django. This wasn’t something I had counted on when I originally made the time line for Mozilla using Pootle but it was a necessary delay. During the transition, forward progress, at least on the Mozilla side, was halted. In November and December, the translate.org.za team did some fantastic work and completely replaced jToolkit.

Thanks to a lot of work from everyone and a bunch of unit tests the django based system reached parity with the old system rapidly. The Pootle team is expecting to release a new version around the end of this month. At that time I’ll upgrade our alpha version and re-enable the features I’ve had to disable. I’m expecting the upgrade to solve a lot of the scalability problems we’ve been having and then we can start advertising our install more and expanding the projects it works with.

Once I do the upgrade Mozilla will be running a stock version of Pootle which I expect to continue from this point forward. Any patches Mozilla contributes back will be generic enough to be useful to anyone and will land on trunk.

We’ve created a 2009 idea/goal wiki page which will be distilled into a project road map. There are some exciting features coming down the pipeline, bringing a lot of improvements (particularly with the user interface) with them. As an added bonus, the new Django framework will allow us to progress faster with new features and it will be easier for more people to contribute code.

Thanks for your patience.

This store will let anyone upload shirt designs and print custom Mozilla shirts. If you’ve got good taste but designing isn’t up your alley you’re welcome to browse and customize the existing designs as well.

I don’t think there are any localized shirts on the site yet but I’m sure they’ll start popping up soon – and once one is designed anyone in your local community can order it. Zazzle, the company we’re using, says they ship to 84 countries which isn’t everyone but it’s a good start.

I’m excited that localized shirts are finally available and I’m looking forward to some great designs.

The original plan for Verbatim was to branch Pootle and continue to merge code between the branch and trunk as features were developed. As we developed code we realized this just creates more work for developers and makes users have to choose between two branches. A better method is just to have everyone commit to trunk and write the code in such a way that any site specific code is maintained in configuration files which is what the current plan entails.

Step 1: Branch Pootle. Wynand created the Mozootle branch as a place for Mozilla developers to commit. For the record, Verbatim is the name of the project, Mozootle is the name of the actual branch in the repository; effectively, they are interchangable.

Step 2: Develop Mozootle. We used Bugzilla to track our changes and there are still plenty to do but the first milestone is closed at this point.

Step 3: Merge Mozootle to Trunk. This is the stage we’re currently at. The translate.org.za team has been reviewing the changes in Mozootle and planning out the merging strategy. There is a fast moving wiki page tracking issues with the merge right now. Our current goal is to resolve these and merge in the next couple days.

Step 4: Continue development on Trunk. The plan after the merge is for all developers to continue committing code into the Pootle trunk. What “development” means is a post in itself so I’ll cut this off here.

In hindsight the whole branching and merging process seemed necessary at the time but it doesn’t feel like we gained much. Next time I think we’ll just skip the branch.

The server is a bit more unstable than I’d like[1] but it’s usable. If you’d like to give it a shot to localize the latest AMO changes follow these steps:

1. Log in with your LDAP account (no need to register separately)
2. Click “change options” and choose “addons.mozilla.org” and your language
3. Join #verbatim on IRC and let me know you registered and I’ll make you an admin for the locale. After that other options will appear in the interface.

One side note:

To update the .po file from SVN or commit your changes to SVN you need to click “Show Editing Functions” when viewing the LC_MESSAGES folder.

I’m coming up with a plan to make both the registration process simpler and the updating/commiting more intuitive but for now that’s what we’ve got. If you’d prefer to keep updating the way you’re used to feel free; this is an alpha version and I’m primarily seeking feedback (and hopefully offering some convenience).

If you’re interested in learning more about Verbatim there is information on the wiki including the current time line in the meeting notepad section.

[1] Due to the way jToolkit works I have to restart the server every time I push a code update which breaks everyone’s sessions.