Skip to content

Small teams and the Marketplace workflow

Towards the end of last year my routine was to find a weeks worth of bugs for each Marketplace developer, assign what I thought they could do in a week, and then meet with the person individually to talk about them and make sure we both thought it was a reasonable amount and the priorities were right. This worked fine for smaller teams, however, it’s not unusual for us to close 70+ bugs a week these days and it simply used too much time. I’d often end up skipping the folks who were comfortable grabbing bugs on their own, which was fine, but it didn’t encourage discussion within the team of who was working on what and kind of pushed people into working in isolation.

The last time the whole team was in the same room together (six months ago, now?) I proposed a new plan which we’ve been following ever since. The plan recognizes that the components within Bugzilla already split the Marketplace along logical lines (eg. API, Statistics, Reviewer Tools, etc.). Using those components as a template I asked everyone what areas they were most interested in and organized small groups which were responsible for each component. These groups meet once a week to do three things for each component:

  1. Review any new bugs in the component since last time and set priorities.
  2. Make sure the bugs people are working on are the most appropriate and that they are in the current milestone.
  3. If anyone is blocked on anything either fix it or escalate it.

There are generally only a few new bugs in each component each week so the meetings are often only a few minutes. These meetings make discussing current features a part of the natural development flow instead of making developers go out of their way to find out what is happening. If anyone is stuck or confused it’s easy to speak up and get help and it also gives anyone who is interested in that part of the Marketplace a place to go to ask questions (meetings are open to anyone).

The main concept behind the plan is that developers are aware of what our goals and commitments are and know better than I do what needs to be done to get there. If I’m in between them and closing bugs I’m just in the way. This plan empowers developers to be owners of their components and features and more officially shares the experience of triaging, prioritizing, and breaking specs apart into bugs.

I still go to as many of the component meetings as I can and help push things in the right direction but I have confidence that if I have a conflicting meeting or am focused on something else I can skip the meeting and there won’t be any critical bugs that don’t get addressed. Using the system has allowed us to add several more people to the team without having additional layers of management and still allows me to focus on coordinating with other teams at Mozilla instead of spending all my time on just the Marketplace team. Additionally, I still look at every bug that is filed for the Marketplace, but I’m routinely a couple days behind on generic bugmail. If a blocker bug is filed it is usually caught right away with multiple people watching their components.

The groups aren’t set in stone and several people have already switched around. They are simply a starting point – if nothing else, a bucket of bugs developers can go to when they have a couple hours to fill and they don’t want to spend their time sorting through our entire list of open bugs.

After several months of using the new system I’m happy to say that it has taken great strides towards solving the two most common concerns the marketplace team had: What are my teammates working on, and what am I working on next week/month? Our strategies will continue to evolve as we grow and change but this has been an effective change for where we are at right now.

Tagged , , ,

The sunset of

Over four years ago was built as a gallery to hold Personas – easy to make and use themes for Firefox. A lot of programs were supporting skins at the time, but I don’t know of any which were so simple (literally, move your mouse over an image on a webpage and it’s applied). A community of artists formed and many designs emerged – some amazing and popular, some focused on specific things (hat tip to CollieSmile), many for sports, or brands, or just people who wanted pictures of their family in a spot they noticed all day long.

Honestly, I thought they seemed gimmicky at first, but I started using them casually and they can be pretty fun. Every once in a while I’ll look up at my search box and see this guy and crack a smile and that’s enough reason to keep using them for me.

Now I’m treading down the dangerous path of naming names – a road full of mine shafts and sharp turns – and if I make a mistake, my apologies, please let me know or leave a comment. I wasn’t involved at the start of the Personas project but I think credit for the idea goes to Chris Beard, original add-on development to Myk Melez, original Firefox development work to the Mozilla Labs team, Toby Elliott and Ryan Doherty as the primary web developers, Stephen Donner and Krupa Raj for QA, and Jeremy Orem for IT Support. I know other teams worked on code here and there, but I think those were the primary contributors. I was focused on the engineering aspects so I’m not sure who to credit for the operational side of things – I know someone was managing day to day life at the time (it was Deb Richardson at some point, and today it’s Amy Tsay) and there were numerous volunteers reviewing submissions as they came in. It’s probably worth noting that in all the queues of all the sites at Mozilla the queue was the only one I regularly saw completely finished and empty – kudos to those reviewers.

After seeing the rapid success of Personas we began talking about what we should do with them. The site didn’t have anyone adding features or doing maintenance on it – Ryan was the closest person to it and officially he was working on AMO. So, after some discussion we decided we would migrate all the data in to AMO which would keep all of our themes and add-ons in one place. We finished the first steps (relatively) quickly – we wrote a questionable script which ran periodically and synced Personas listings from to AMO. The files themselves and all the users remained on Looking purely at the number of times the script finished vs failed, it did pretty well, but there were times when it would fail constantly until someone could find time to fix it, much to the concern of the artists who didn’t see new data showing up on AMO.

Meanwhile, our security team decided to add to the bug bounty program which surprised everyone and which we’ve playfully chided them about ever since. Some of our oldest and hacked together code (remember, this was originally an experiment) was now coming under scrutiny from security professionals and bored/gifted students from around the world. Special thanks to Matthew Noorenberghe and Brandon Savage for writing patches to fight back the flood of incoming security bugs.

A final spur to getting the migration finished came in the form of Mozilla Persona – an aptly named identity system, also from our company. After some spirited debate it was decided the identity project would use the Persona name and the personas on would be referred to as Themes (and the existing themes we had as Complete Themes). With all the logistics out of the way we rapidly adopted the new names everywhere save which continued to cling to life.

Late last year Chris Van Wiemeersch spearheaded a clandestine operation to finally finish closing down (with special mention to Kevin Ngo for contributing piles of code all the while finishing his final exams, and Kris Maglione for updating the Personas Plus add-on).

After another quarter of late nights balancing the migration with developing the Firefox Marketplace it’s finally happened: I’m happy to announce that has now officially retired and all the content and artist accounts have been migrated to AMO.

It’s exciting from a Mozilla point of view because the confusion of projects can finally end and we can stop pretending to maintain a site that we haven’t actually done anything with for years. The artists and users have reason to be happy also, though – moving to AMO means it’s on an actively maintained site. Improvements can happen, features can be added, and new ideas can be explored. Finishing this long awaited migration will breathe new life into the themes community and I’m looking forward to all the fun new ways we’ll see Perso-uhh – I mean – themes, being used.

A screenshot of new users on AMO on the night we imported the accounts. Our usual numbers of 1500/day are eclipsed by 1.5 million that night. Also a testament to the site’s scalability that it didn’t miss a beat.

Tagged , , ,

Inception: projects within projects

When we converted from PHP to Python I mentioned how deceptively large (lines of code) the site had grown with so many views and features. We’ve since built on literally the same code base (a decision I hoped to blog about some day, alas) and the line count continues upwards. The MVC paradigm is one of the most common out there, easy to understand, and built in to our frameworks so it was naturally a good choice. However, as the site continues to expand we’ve had to review what the best architecture for the code would be and the main point that stands out is that our entire site and its services are a monolithic blob of code. Both sites, all of our APIs, all of the PayPal code, the administrative tools, the developer control panel, the reviewer queues, the list goes on – it’s all in one repository and changing one means potentially affecting all of them.

With big plans for 2013 and no signs of slowing our code expansion we’ve hit the point where it makes more sense to break the site up into smaller projects. We’ve already started with our payment processor and metrics processor and more are soon to follow. These projects will be tied together around a central API on the back end. We’ve had partial front end APIs for years (like search) which we will expand to make the entire site accessible via API. At that point our front end code will be completely separated and can be optimized to speak directly to JSON APIs instead of pulling HTML with every request (something which the developers have been demoing the past couple weeks at our show and tells). A rough diagram is below, subject to change, of course.

Marketplace Architecture Diagram

Some benefits of this approach include:

  • Physical separation of code which will easily allow us separate SLAs or even separate conformance agreements, like PCI compliance
  • Speaking to APIs means we’ll have specific failure cases we can write code for so when a piece of the site dies the entire site doesn’t crash
  • Each project can be written to scale horizontally without affecting the other projects
  • Developers and contributors can focus on a single project without having to check out the entire site (and deal with substantial configuration maintenance)
  • Localizations will be broken up into separate .po files, a long requested feature from our localizing friends
  • Deployments can happen for specific projects without updating all of them. These will be faster, easier to scope, and easier to roll back if necessary
  • 3rd parties can build full featured sites off our APIs

There are a couple of downsides: maintenance/configuration of each project and integration testing. The former is diminished a bit because not every developer works on every project so with some prudent mocking and a few virtual machines there should be an easy flow there. The second is something we currently neglect but will become critical with these changes so expect some solid developments there. With all the other work on our schedule I think we’ll be touching parts of this project in every quarter this year. If you hear the codename INCEPTION it’s referring to this project.

Tagged , , , , ,


A man boarding a train

The situation had soured. Tony Six and his goons had known I was coming for four hours which means anyone that owed him a favor or had a score to settle with me was already on their way and settling in to wait. It wasn’t the kind of party I liked to crash but you don’t always get to choose the venue. I had one ace they wouldn’t be expecting. They didn’t know I’d have a train.

Tagged ,

Mozilla t-shirt time capsule

I moved around a lot over the past year and ended up putting a bunch of old t-shirts into a trunk and forgetting about them. My one-o-clock-in-the-morning brain decided now was a good time to go through the trunk and get rid of all those old shirts I never wear. It turns out we Mozillians used to really like making t-shirts. Below is a pile of shirts I found from around 2005-2011 or so. How many can you identify?

Mozilla Shirts


A successful first FLOSSHack

A few months ago Tim Morgan emailed the Portland OWASP chapter and suggested that we organize a meeting where everyone could get together and audit some existing software. When vulnerabilities were found we would follow the responsible disclosure life cycle and notify the maintainers before publicly disclosing. It would be a fun way to spend an afternoon, people would get experience with responsible hacking, and the software maintainers would have the opportunity improve their software. The idea gained traction and FLOSSHack One was born.

I coordinated with Ushahidi, an open source mapping application, to be the subject of the audit. They were very helpful, even participating during our meeting and answering questions about how their application was built. We had about ten people show up on a Sunday afternoon and everyone got to work in a supportive environment for penetration testing. By working with the developers we were able to cover a lot of the code and came up with several vulnerabilities to report which will be announced on as appropriate.

Thanks to Tim for organizing the FLOSSHack, Ushahidi for supporting us on their end, and thanks to everyone who participated. If you’re interested in doing your own, Tim wrote this generic wiki page which should help.

Tagged , , ,

Adding a debug language to ȧḓḓ-ǿƞş.ḿǿzīŀŀȧ.ǿřɠ

Last week Greg Koberger finally got me to cross “add a test locale to AMO” off my list – and it turns out it only took a few minutes of actual coding. It sounds like others have had some troubles so I wanted to run through what I did.

Firstly, you can see what I’m talking about at the AMO dev site. I’m using a script in the Translate Toolkit called podebug to generate a .po file filled with unicode characters. You can also generate one which surrounds the strings with “xxx” or blanks them out, etc. The goal is to easily see what strings on your website can’t be localized (this means they aren’t in the .po files for localizers).

If you’re using product-details to get locale names make sure you’re using the latest revision – I added dbg to the array so it will show up in language dropdowns like all the other languages. After that I added dbg to Django’s LANGUAGES define[1]. At that point the language showed up, but we’d get tracebacks if we tried to format money in a string since ‘dbg’ was a fake language. AMO uses Babel for details about each locale so I just made a wrapper for the Babel Locale() function and told it to use ‘en’ if it saw ‘dbg’. This way ‘dbg’ is used for all the strings on the site but for any complex formatting we fall back to en. This works since dbg is based off en anyway, as far as pluralizations are concerned. You can see my commit which added that function and made sure we started using it here.

This also works on the marketplace.

[1] I actually added it to AMO_LANGUAGES which gets to LANGUAGES in a slightly round-about way. See our file for how that flow works.

Tagged , , , , ,

How to get a development instance of AMO set up in about 10 minutes

Last year we set up landfill.amo to give contributors an easy base to set up the site. Easy is relative here, of course, but it was a big leap over what we had at the time.

Kumar leapfrogged that milestone by adding Vagrant configuration scripts to our repository. Now you can have a running version of AMO on your local system in about 3-5 commands[1].

Check out the steps to install AMO with Vagrant to see how. I set it up on OS X last week and aside from waiting for the download it only took a few minutes. Drop by #amo on IRC if you run into any troubles.

[1] Setting this up on Windows is apparently more difficult although one contributor did find success after fighting with it for some time.

Edit: Updated the installation instructions URL

Tagged , , , ,

10 years of Irssi use and I switched to WeeChat last weekend

I started using Irssi almost 10 years ago when I first started trolling wandering around the world of IRC. My main use is to run it in a screen and stay connected all the time. To chat I’ll just ssh into the server and reconnect to the screen. Generally I leave my terminal open, stretched across the top of my monitor so it’s really wide but only about 8 or 10 lines high. This way I can keep IRC visible all day long and respond quickly to questions.

I’ve customized irssi with custom highlights and commands and all has been well, save one thing: bug 310 – vertical splits. Irssi can’t do vertical window splits and with the trends giving us widescreen monitors, horizontal splits aren’t really useful to me. The bug has been open since 2005 and I’ve all but given up on it.

Then the other day Chris McDonald claimed the unfortunately named WeeChat was superior to Irssi. I was all ready to defend Irssi’s honor but I read through the documentation and WeeChat was pretty compelling – most notably its support for vertical splits!

So, long story short, I switched last week and it’s awesome. I don’t miss Irssi at all and in fact WeeChat offers me things I never even knew I’d want (like per-buffer history when hitting the up-arrow). I’ve customized it so it essentially looks like Irssi (no nickname list, etc.) and I’ve added a lot of aliases to make it easier for me to use (mostly vim keys). If you’re an Irssi user you owe it to yourself to at least read their docs and see what you think.


PHP is dead! (on

This is just a short note to recognize the long coming milestone of PHP being effectively off[1] on We started the migration in 2010 and just finished it up a couple weeks ago. After the major pages were completed it was hard to budget time for all the minor details we had implemented since there was so much other important stuff to do (I’m looking at you, Now that the switch is done though we can simplify our setup instructions for AMO, simplify our infrastructure, optimize apache for python only, have full unit test coverage – the list goes on.

A big thanks to all the developers who made the switch possible, and especially the ones at the end who were working on migrating PHP scripts instead of more glorious projects. Thanks to the management and all the other folks affected by the switch for being patient with the scheduling. Thanks to the localizers for dealing with the crazy merged .po files, and, of course, thanks to the users of AMO for reporting bugs when they happened and generally being an all around great community to work with.

[1] Currently all traffic is being rewritten to a WSGI handler for Python. PHP is still on the server but nothing uses it. We’ll be removing it completely in the near future.

Tagged , , , , , , ,