I attempted this tonight and it was not a pretty upgrade (or install, for that matter). Among the medley of problems is Django ticket #6548. Django assumes it’s not behind an SSL proxy so when it does any redirects it doesn’t use https. This means logging in and logging out work on our server but the user is presented with a jarring “bad request” interstitial.

The current status is that user accounts are not migrated and, even if they were, I can’t seem to set permissions for projects. Since there are some odd problems that we haven’t seen elsewhere and this is an alpha install I’m going to leave it as is and debug some of the issues over the next few days. Expect downtime. If there are questions visit #verbatim on irc.mozilla.org.

While writing code it became apparent that the framework Pootle was built on, jToolkit, had some shortcomings that were making it difficult to work with (not to mention development had been stopped on it since 2006). The decision was made to migrate the back end of Pootle from jToolkit to Django. This wasn’t something I had counted on when I originally made the time line for Mozilla using Pootle but it was a necessary delay. During the transition, forward progress, at least on the Mozilla side, was halted. In November and December, the translate.org.za team did some fantastic work and completely replaced jToolkit.

Thanks to a lot of work from everyone and a bunch of unit tests the django based system reached parity with the old system rapidly. The Pootle team is expecting to release a new version around the end of this month. At that time I’ll upgrade our alpha version and re-enable the features I’ve had to disable. I’m expecting the upgrade to solve a lot of the scalability problems we’ve been having and then we can start advertising our install more and expanding the projects it works with.

Once I do the upgrade Mozilla will be running a stock version of Pootle which I expect to continue from this point forward. Any patches Mozilla contributes back will be generic enough to be useful to anyone and will land on trunk.

We’ve created a 2009 idea/goal wiki page which will be distilled into a project road map. There are some exciting features coming down the pipeline, bringing a lot of improvements (particularly with the user interface) with them. As an added bonus, the new Django framework will allow us to progress faster with new features and it will be easier for more people to contribute code.

Thanks for your patience.

The original plan for Verbatim was to branch Pootle and continue to merge code between the branch and trunk as features were developed. As we developed code we realized this just creates more work for developers and makes users have to choose between two branches. A better method is just to have everyone commit to trunk and write the code in such a way that any site specific code is maintained in configuration files which is what the current plan entails.

Step 1: Branch Pootle. Wynand created the Mozootle branch as a place for Mozilla developers to commit. For the record, Verbatim is the name of the project, Mozootle is the name of the actual branch in the repository; effectively, they are interchangable.

Step 2: Develop Mozootle. We used Bugzilla to track our changes and there are still plenty to do but the first milestone is closed at this point.

Step 3: Merge Mozootle to Trunk. This is the stage we’re currently at. The translate.org.za team has been reviewing the changes in Mozootle and planning out the merging strategy. There is a fast moving wiki page tracking issues with the merge right now. Our current goal is to resolve these and merge in the next couple days.

Step 4: Continue development on Trunk. The plan after the merge is for all developers to continue committing code into the Pootle trunk. What “development” means is a post in itself so I’ll cut this off here.

In hindsight the whole branching and merging process seemed necessary at the time but it doesn’t feel like we gained much. Next time I think we’ll just skip the branch.

When I started the planning for branching Pootle I was very focused on scalability (or lack thereof) and most of my initial goals were to improve that including replacing flat files with mysql, creating a cacheable URL structure, etc. In hindsight, I should have realized that this project wasn’t going to be getting nearly the traffic load some of our other sites were getting and my priorities were out of order. What I should have been thinking about was usability and interface improvements. Due to my lack of foresight the project launched with enhancements in both areas but I think the time we spent on scalability was premature and the user interface suffered.

Whether it’s writing more comments than code or making sure meetings have agendas I’m a huge fan of communication. When branching a project, particularly when there are plans to merge the branch back into trunk, communication is vital. I think our meetings are productive but communication on a smaller scale is still a struggle. Both Pootle and Verbatim ended up writing the same code in a few cases which could have easily been avoided. In this particular case the timezones can make it difficult to synchronize but it’s something I’ll work at more in the future.

Something we have done a good job with is making a schedule and updating it with new developments. I really want to expand our effort here though. I think one of the difficulties of someone joining a project like this is direction; what are the goals of the project and how are we getting there? Several of us have talked about it on IRC and we all have a good general idea but for someone that isn’t as involved it’s hard to follow. Once we get over the next big hump (replacing jToolkit) I think this will begin to fall into place with smaller bugs/features revealing themselves and providing a way for volunteers to get footholds on the project as a whole.

Lastly, it might be obvious, but if you’re planning on maintaining the branch or merging back to trunk make sure you get along with the lead developers. I’m fortunate to work with the Pootle developers who clearly care deeply about the project. From talking to them it’s obvious they have the end users’ best interests in mind and are excited that we can all work together to improve the end product. And really, that’s what open source is all about and it’s great to be a part of it.

The server is a bit more unstable than I’d like[1] but it’s usable. If you’d like to give it a shot to localize the latest AMO changes follow these steps:

1. Log in with your LDAP account (no need to register separately)
2. Click “change options” and choose “addons.mozilla.org” and your language
3. Join #verbatim on IRC and let me know you registered and I’ll make you an admin for the locale. After that other options will appear in the interface.

One side note:

To update the .po file from SVN or commit your changes to SVN you need to click “Show Editing Functions” when viewing the LC_MESSAGES folder.

I’m coming up with a plan to make both the registration process simpler and the updating/commiting more intuitive but for now that’s what we’ve got. If you’d prefer to keep updating the way you’re used to feel free; this is an alpha version and I’m primarily seeking feedback (and hopefully offering some convenience).

If you’re interested in learning more about Verbatim there is information on the wiki including the current time line in the meeting notepad section.

[1] Due to the way jToolkit works I have to restart the server every time I push a code update which breaks everyone’s sessions.

On the SVN server the first thing we did was to create a special Verbatim user that can commit to SVN via SSH using a generated key. We copied this key to the Verbatim host which allowed us to commit as the verbatim user without typing a username or password.

The only thing that was added to the Verbatim code was a patch that Dan Schafer cooked up that sets an SVN revision property, translate:author, to the name of the current user. When the user clicks “commit” this property is set and sent along with the commit.

At this point we could commit from the application but it still goes to the application as the Verbatim user. We used SVN’s hooks to take the next step.

The first script we changed was the pre-revprop-change hook. This controls what special revision properties a user can modify when they commit. Our script adds the ability to modify svn:author and translate:author. Before allowing the modifications the script checks if the user committing is the special verbatim user to prevent anyone from committing as someone else.

Next we added a post-commit script that looks for the translate:author property. If it’s found it will take that value, replace svn:author, and remove translate:author; effectively making whatever was in translate:author the real author. This is a non-versioned change which means there is no commit that needs to happen – the new author is set immediately.

With these scripts in place we can commit as anyone from the application and everyone’s credentials stay encrypted and secure.

The localization tool we’ve decided to use for Mozilla projects currently works solely on standard .po files. Dan Schafer wrote a conversion script that will allow us to convert from the AMO .po style to the standard gettext style. In order to create a standard .po file that can be easily localized we’ve decided to use a keyword we haven’t used before: msgctxt.

msgctxt allows us to store the context of a phrase without affecting the actual phrase. The script Dan wrote combines two AMO .po files, the English version and the target locale file, into one. For example, take these two files:

The English AMO .po file:

msgid “home_header_greeting”
msgstr “Hello!”

The French AMO .po file:

msgid “home_header_greeting”
msgstr “Bonjour!”

And combine them into a single .po file containing the following:

msgctxt “home_header_greeting”
msgid “Hello!”
msgstr “Bonjour!”

We haven’t used the msgctxt keyword until now because it’s as of yet unsupported in PHP but editing tools that support gettext 1.5 and above should be able to edit the files with no trouble. We’ll still be using the AMO style .po files in SVN and in the AMO product itself so if you’re used to the AMO style and you plan on continuing to commit directly to SVN it shouldn’t affect your work flow.

If you’d like to use the combined .po style outside of Verbatim there will be a way to import and export the file format. If you do edit the file with an external editor please be careful not to change the context of a string or your changes won’t be saved.

As always, we’re happy to hear questions or concerns. Look us up in #verbatim or leave a comment here if you want to get in touch with us.

• Connecting localizers and add-on authors. The only way to localize add-ons and metadata right now is if the author knows someone who can translate the extension or by using an existing service like BabelZilla. It would be nice to add an easy way to connect these two without overwhelming localizers or creating a bottleneck.
• Being notified of AMO string updates. Right now we have RSS feeds and the mailing list. It would be nice to have something more concrete and consistently up to date.
• Use normal .po files or make the ones we have easier to use. This came up before and was discussed again briefly. I don’t expect to convert to using standard .po files now that we can provide English fallback in the application. However, we can make them easier to use particularly with the small (size and number) of updates for AMO.

What other ideas and pain points would you like to discuss at the summit? Leave a comment with ideas.

Also, I expect Verbatim to come up in our discussions since it’s still in planning and now is the time to add features. If you have ideas for AMO in a Verbatim context feel free to add them also.

We’ve decided to use the codename Verbatim to refer to this project.

I’ve started some wiki pages that are short but growing. As usual there is a meeting notepad which anyone is welcome to peruse. The wiki page offers a brief description but I can give a short summary of our current situation and where I’d like to see it go. This tool will be used for both the Firefox browser and Mozilla web sites. I expect my posts to lean towards talking about the web since that’s where my brain is but it’s important to keep both in mind.

The current situation:

• L10n projects are scattered around in different places. Most web sites are in SVN but the browser’s L10n is in CVS.
• L10n projects use several different formats.
• There are unnecessarily complex permissions in our system. Localizers need an LDAP account and then have to be specifically granted permission in the SVN directory to commit. Sometimes those two get decoupled and strange permission errors happen.
• Some of our projects use a potentially slow process. For example, on AMO, some localizers still add updated translations to a bug as attachments and then I commit them to SVN manually. This takes time and if I don’t get to it for a while the localizer is stuck waiting on me to update before they can verify the changes.
• When we add or change a string on a web site we generally just email the dev-l10n-web mailing list which is easy to miss. There is an RSS feed coming out of SVN but that’s still not an ideal notification system.

The plan:

• Provide a central point of localization where someone can see all L10n projects for a locale and the amount of work to be done on each.
• Provide a common interface for localizing all of our different file types.
• Provide an easy way to change the strings online[1].
• Provide an easy way to be notified of new and changed strings.
• Optionally, provide the public with a way to suggest changes to existing strings. L10n leaders will be able to approve or reject these suggestions at their convenience.

After reviewing many tools online, in the interest of time, completeness, and alignment with the plan, the webdev and L10n-drivers team have decided to use Pootle and the Translate Toolkit for our initial release. Pootle in it’s current form already implements the basics of Firefox L10n and supports regular .po files. The webdev goal over the next few weeks will be to setup an official Pootle development area and start figuring out how to convert our L10n projects’ file types to something Pootle can import and export.

I’m hoping these changes will encourage new users to get involved and make things easier for our long time contributers. The plan is still in development and, as usual, input is welcome. Real time discussion is easiest in #verbatim on IRC. Email and comments here are effective as well.

[1] Just to be clear, I don’t want this tool to become a bottleneck or hinder localizers that have a good process down. It will still be possible to do check-ins to the repositories manually without using the web interface. The current Pootle web interface can be played with or you can watch a screencast for more details.