Last March there was some feedback from developers about being tired of the code they were working on and eyeing other technologies but not having the time to work with them. This is a fragile balance which every manager and developer has had to struggle with. Writing new code is always sexier than maintaining old code, but the old code is the bread and butter that keeps you in business. Finding a happy medium is a noble, elusive, and unfortunately, shifting goal.

Enjoying what you work on is crucial to being happy and productive though so I’ll continue to pursue that goal.

One of the changes we experimented with at the time was to let developers choose their own focus for the quarter. We still had high level quarterly goals that needed to be done, but that left plenty of other time throughout the quarter to work on any of those areas which were all equally important but perhaps not equally as interesting. I filled up a whiteboard with ideas one morning (and left space for developers to add more) and we had a meeting later in the day where anyone could discuss the ideas and sign up for what was interesting to them. This was a short term commitment from the developers and they got to work on what they cared about. From a managerial perspective, it increased motivation but didn’t sacrifice accountability. Since I knew the focus of the developers early in the quarter, I could help clear out roadblocks that they’d meet before they even started working on their areas.

The quarter is over this week and overall the idea was a success. All the feedback from developers was positive – if you’re looking for a way to spice up the top-down approach to goal setting this was effective. It also turned the goal setting into more of a discussion rather than an edict that just shows up.

I wrote this almost 9 months ago but apparently never hit publish. So, here it is. We’ve played with it a bit since but haven’t had a large quarterly planning meeting like that because we’re mainly focused on the marketplace. Experimenting with smaller goals is a next step here, and doing the meetings on a weekly or biweekly basis.

In January of 2010 we started migrating addons.mozilla.org from CakePHP to Django. It was a controversial decision. Developers were ambivalent to excited, managers were opposed to neutral – a split anyone would expect. When I first talked about it I expected to be able to turn off PHP by the end of the year. It didn’t turn out quite like that.

Fifteen months later we’re still transitioning and it’s still stressful. The toughest part about a major migration like this is that there is only one team that is doing the migration, continuing to add the new features we need, and all the while maintaining the old site. That’s a stressful environment for developers since the interactions between the languages can be complicated, it’s stressful for managers because features take longer to complete, and it’s stressful for users (and QA for that matter) because issues will arise which are hard to reproduce and complicated to explain.

In the midst of all the work of migration, the rest of the company is still working: the security team is announcing bounties on our site which means we need to be vigilant about fixing issues, project management continues to come up with features to be added, the site perseveres in its never-ending quest for a new look and feel, and Firefox 4 is using AMO like never before meaning approaching 10,000 hits per second is a regular day. All of that is specific to the add-ons site, but consider your own company if you’re thinking of going down the same road – what is coming up for your site that will throw a wrench in the works?

The meat and potatoes of it really comes down to: Given the hindsight of today, would the migration be a good idea? There isn’t a right answer for every site, but for AMO we did the right thing[1]. As of today the majority of pages that matter are on Python – there are some admin tools, and some cron jobs, and the occasional semi-obsolete public page that is on PHP, but for the most part, we’re looking really good (less hand waving, more real data). My new (overly optimistic?) plan is to have PHP off by the end of this year. We’ll see.

To give you an idea of man-hours, we’ve had anywhere from 3 to 6 superhero developers working on the site over the past 15 months, and it’s looking like the whole thing will take around 24 months. That’s a big chunk of time for a site that needs to grow and evolve as quickly as popular sites do these days.

So, overall, I think the lesson is: any reasonably sized site is going to have rabbit holes in it. At first glance AMO might look like it’s got a dozen “main” pages, with a couple dozen more supporting pages (and throw in a few more for the admin CRUD). Have a look at that spreadsheet I linked above and you’ll see that’s not even remotely the case. The spreadsheet even ignores sub-pages in a few places and doesn’t include any new features added in the past year. If you’re considering a migration, think it through well. Make a spreadsheet of every URL, identify the complicated areas, and make sure everyone is clear on the timeline and what it means for new features. People will absolutely try to scope creep your migration – make it clear if a section of the site is migrating as-is or can be migrated and redesigned at the same time. Redesigns add complexity for the developers but can earn you some good will with the users and managers and if you’re in this boat you can use all the good will you can get.

May you have the best of luck with your decisions.

[1] I’ll write another post about pros/cons of the actual frameworks and platforms. Let’s just assume we’re happy with the technical side of the switch for now.

Migrating from CakePHP to Django

This is a big undertaking and we’ve been discussing it for quite a while. We’re currently the highest trafficked site on the internet using CakePHP and along with that we’ve run into a lot of frustrating issues. CakePHP has serviced AMO well for several years, so it’s not my intention to bad mouth it here, but I do want to give a fair summary of why we’re moving on. Please also note that AMO is still running on CakePHP 1.1 which is, I think, a year out of date? Three substantial issues:

• Useful Database Abstraction Layer: CakePHP has a concept of database abstraction, but we didn’t find it powerful enough. When it did work it would return enormous nested arrays of data causing massive CPU and memory usage (out of memory errors plague us on AMO). When it didn’t work, we’d end up doing queries directly which kind of defeats the purpose. We couldn’t use prepared statements so we’d have to escape variables ourselves. There was no effective caching built-in and since we just had huge arrays as a response there was no effective way to invalidate the cache we were using (see: Caching is easy; Expiration is hard). The DB layer should return objects that are easy to cache and easy to invalidate. The built-in Django database classes (combined with memcache) should work fine for us here.
• Effective unit tests: I’ve beat the drum about our unit tests before but the simple matter is that it’s really difficult to do them right with the tools we are using. Our test data is already very limited, but if we try to run all our tests right now they’ll run out of memory (and take forever). The CakePHP method of mocking controllers and models was inadequate for what we needed and difficult to deal with. We want our unit tests to run quickly, from the command line, and be independent from each other so there aren’t intermittent problems to waste our time with. We’ll be using Django’s built-in testing framework.
• Better debugging: Debugging in CakePHP amounts to defining a DEBUG level and seeing what is printed on the screen (usually the giant arrays). We supplemented this with Xdebug where we needed it, but that’s still not enough. A framework should have excellent logging and on-the-fly debugging that displays a full traceback (often something will fail deep within CakePHP and we’ll get the file/line where PHP gave up, but not the line in our code that started the problem), the values of variables, the page headers, server settings, SQL that was run, what views and elements are in use, etc. We’re planning on using a combination of pdb, IPython, and the django-debug-toolbar to make all of this easily accessible while developing.

Those are the major issues we’re having right now, but if you want to dig into the comparison some more check out our discussion wiki pages, but realize the majority of discussion happened in person.

Moving away from SVN

We moved AMO into SVN in 2006 and it’s treated us relatively well. Somewhere along the line, we decided to tag our production versions at a revision of trunk instead of keeping a separate tag and merging changes into it. It’s worked for us but it’s a hard cutoff on code changes, which means that while we’re in a code freeze no one can check anything in to trunk. As we begin to branch for larger projects this will become more of a hassle, so I’m planning on going back to a system where a production tag is created and changes are merged into it as they are ready to go live.

Most of the development team has been using git-svn for several months and, aside from the commands being far more verbose, we haven’t had many complaints. We’ve discovered Git is a much more powerful development tool and we expect to use it directly starting some time next year. As of now, we expect to maintain the /locales/ directory in SVN so this change doesn’t affect localizers but we’ll keep people notified if there are any changes to that process.

Continuous Integration

I mentioned excellent testing being one of the reasons we’re moving to Django. Along with that testing is the opportunity for continuous integration. We plan on using Hudson as the framework for our continuous integration. With excellent test coverage and quick feedback from Hudson this should drastically lower our regressions and boost our confidence when we deploy. Speaking of which…

Faster Deployment

For most of 2009 we’ve pushed on 3 week cycles. 2 weeks of development, 1 week of QA and L10n. Delays and regressions being what they are, I think we averaged a little better than a push a month. This is a fairly rapid cycle for a lot of development shops, but I feel like it’s holding us back. We’ve heard a lot of success stories about shorter cycles and I’d like to aim for deployment (optionally, of course) of a few times per week. By shortening the development cycle we reduce the stress of:

• the developers: Everyone likes to see what they’ve done go out quicker and it means less conflicts with others when the patches are smaller.
• the QA team: Right now we dump 2 weeks of work on them and say we need it done right away. With smaller cycles they can verify small changes as they go and not be overwhelmed.
• the infrastructure team: Smaller changes means less to go wrong and with a continuous integration server and some automation they can have minimal involvement with the whole process.
• the localizers: Every time we release we dump a bunch of changes on these fantastic people and tell them we need them back in a week. Most of the time they plow forward and get them done on time. If they don’t though, they are stuck with waiting for the next 3 week cycle. If we push often, it’s not a big deal.
• the product managers: These guys come up with crazy ideas for us to implement and then they stare at graphs and numbers to see if it worked. With shorter cycles they can get faster feedback about what works and what doesn’t.
• the users: Faster release cycles means bugs that are fixed in the repository are fixed on the live site sooner. ’nuff said.

Process Data Offline

Much of AMO relies on cron jobs to get things done. All the statistics, add-on download numbers, how popular an add-on is, all the star rating calculations, any cleanup or maintenance tasks – these are all run via cron and they are so intensive that the database has trouble keeping up. We’re planning on utilizing Gearman to farm all this work out to other machines in incremental pieces instead of single huge queries. Any heavy calculating that can be done offline will be moved to these external processors which should help improve the speed of the site and make all our statistics more reliable (as currently the cron jobs have a tendency to fail before they are complete).

Improve the Documentation

Documentation is a noble goal of many developers but it rarely gets enough attention. We evaluated our current documentation and found it is woefully out of date. By being on a wiki that is rarely used it doesn’t get updated except when someone tries to use it and sees it’s not right. We’re hoping to change that by moving the developer documentation into the code repository itself. We’ll be able to integrate with generated API docs, style the docs however we want, and check in changes right along with our code patches. When someone checks out a copy of AMO, they’ll get all the documentation right along with it. We’ll use Sphinx to build the docs.

The outline above details several large, high-level changes but there are a lot of other plans for smaller improvements as well. This post got a lot longer than I was expecting, but I’m really excited about the direction AMO is headed for 2010. As these changes are implemented the site will become more responsive and reliable, and we’ll be able to adapt to the needs of Mozilla’s users even faster. As always, feedback and discussion are welcome and stay tuned for further back end improvements.

Since that time msgctxt has become a standard part of gettext and makes my substitution string recommendation obsolete. I wanted to officially come out and say: substitution strings are a pain. The scripts we used made it manageable but finding strings in the code meant searching through the .po and not only was this painful for our developers but I think it confused contributors as well.

In our latest release, we’ve converted AMO to use regular .po files now. On the off chance someone followed my advice and would like to convert their site to regular .po files as well, Zbigniew Braniecki wrote a bit about the process and you can grab his scripts (and read about the troubles I had) at bug 501988.

Since I tagged this post hindsight I guess I should look back and conclude something too. Would I do it again? At the time, there were no great alternatives. So, yeah, I would get more opinions on whether the Gnome/KDE method was better, but I would do it again. I think it was the best choice out of several poor ones, but that doesn’t mean I’m not very happy to be rid of them.

[1] To be fair, there was a more common way, used by some KDE and Gnome projects, which was to put a delimiter in the msgid and keep the context on one side and the original string on the other. This is also pretty hacky and you can read about Gnome’s migration away from that method if you’re curious.

The original plan for Verbatim was to branch Pootle and continue to merge code between the branch and trunk as features were developed. As we developed code we realized this just creates more work for developers and makes users have to choose between two branches. A better method is just to have everyone commit to trunk and write the code in such a way that any site specific code is maintained in configuration files which is what the current plan entails.

Step 1: Branch Pootle. Wynand created the Mozootle branch as a place for Mozilla developers to commit. For the record, Verbatim is the name of the project, Mozootle is the name of the actual branch in the repository; effectively, they are interchangable.

Step 2: Develop Mozootle. We used Bugzilla to track our changes and there are still plenty to do but the first milestone is closed at this point.

Step 3: Merge Mozootle to Trunk. This is the stage we’re currently at. The translate.org.za team has been reviewing the changes in Mozootle and planning out the merging strategy. There is a fast moving wiki page tracking issues with the merge right now. Our current goal is to resolve these and merge in the next couple days.

Step 4: Continue development on Trunk. The plan after the merge is for all developers to continue committing code into the Pootle trunk. What “development” means is a post in itself so I’ll cut this off here.

In hindsight the whole branching and merging process seemed necessary at the time but it doesn’t feel like we gained much. Next time I think we’ll just skip the branch.

When I started the planning for branching Pootle I was very focused on scalability (or lack thereof) and most of my initial goals were to improve that including replacing flat files with mysql, creating a cacheable URL structure, etc. In hindsight, I should have realized that this project wasn’t going to be getting nearly the traffic load some of our other sites were getting and my priorities were out of order. What I should have been thinking about was usability and interface improvements. Due to my lack of foresight the project launched with enhancements in both areas but I think the time we spent on scalability was premature and the user interface suffered.

Whether it’s writing more comments than code or making sure meetings have agendas I’m a huge fan of communication. When branching a project, particularly when there are plans to merge the branch back into trunk, communication is vital. I think our meetings are productive but communication on a smaller scale is still a struggle. Both Pootle and Verbatim ended up writing the same code in a few cases which could have easily been avoided. In this particular case the timezones can make it difficult to synchronize but it’s something I’ll work at more in the future.

Something we have done a good job with is making a schedule and updating it with new developments. I really want to expand our effort here though. I think one of the difficulties of someone joining a project like this is direction; what are the goals of the project and how are we getting there? Several of us have talked about it on IRC and we all have a good general idea but for someone that isn’t as involved it’s hard to follow. Once we get over the next big hump (replacing jToolkit) I think this will begin to fall into place with smaller bugs/features revealing themselves and providing a way for volunteers to get footholds on the project as a whole.

Lastly, it might be obvious, but if you’re planning on maintaining the branch or merging back to trunk make sure you get along with the lead developers. I’m fortunate to work with the Pootle developers who clearly care deeply about the project. From talking to them it’s obvious they have the end users’ best interests in mind and are excited that we can all work together to improve the end product. And really, that’s what open source is all about and it’s great to be a part of it.