Skip to content

Improving our process; part 2: Public Priorities

Part 1: Identifying Priorities

As promised, there is now a public view of upcoming work on the Firefox Marketplace. This dashboard shows a rough representation of our thought process when we evaluated each project using the criteria from Part 1. If you hover over the titles there are also some helpful descriptions and questions we asked ourselves. Clicking a project title will expand to show some information about the project as well as any notes about the scoring. Some of the projects have been grandfathered in to this process so they are a bit light on details, but if you look at a project like Support Low Memory Firefox OS Devices (128MB) you’ll see the potential.

We’re still working on the right people to get in the room to do the scoring. Mailing lists are hard because of the lack of easy discussion but in person is hard due to scheduling and not wasting time bikeshedding. We’ll get better as we get experience, our main focus at this point is staying consistent with the scoring – being sure to consider the entire ecosystem when assigning a value.

There are still a lot of projects to evaluate but I’m really excited about being able to link to a backlog of projects and easily point out which are a high priority and why. As always, feedback welcome.

Tagged , , ,

Ten years of addons.mozilla.org

Ten years ago, Ben Smedberg and Wolf landed the first version of the code that would make up AMO. At the end of 2004 it was still called update.mozilla.org, written in PHP, and was just over 200k, compressed.

Behold the home page of update.mozilla.org.

This moved through, I think, 1 major revision “update-beta” which added a database tabled called main that held all the add-on info. This came with some additional fancy styles:

I started around 16 months after this foundation was laid. AMO was essentially flat HTML files sprinkled with PHP code and database queries, just like everyone says not to do now. I think the code was running on a single server named Chameleon at the time and as we released versions of Firefox it would get overwhelmed and become unresponsive. Most of the work at this time was adding in layers of caching like Smarty and keeping everything in memcached.

We had a major rewrite in planning (one with more dynamic possibilities) and I chose CakePHP as the foundation. That release happened and brought another redesign with it:

Check out how those gorgeous pale yellow and faded blue bars complement each other. Also note that we’re still claiming to be in beta at this point.

The Adblock detail page in 2006. Rated 5 out of 5 by someone named Nick!

Finally, at the end of 2006 the infamous chopper design appeared and took over the top third of our site:

Customizability was the message we were trying to communicate with this design. There were some concerns voiced at the time…

This version followed the first quickly. The header was reduced slightly, the content got some more margin, and we started using whitespace to keep things looking simple. On the other hand, you’ll notice our buttons getting more complicated – this was the beginning of our button horrors as we started to customize them for each visitor’s device.

Both chopper designs were relatively short lived and, I want to say around early 2008 we moved to another infamous design – the green candy bar.

Here we are emphasizing search. And how. There was a rumor we actually stole this design from another site at the time – I assure you that wasn’t the case.

I don’t remember the actual date on this one, but let’s say 2009 since we seem to redesign this every year. This was a complete departure from our previous work and the design was done from scratch.

Not a bad looking design and one that served us well.

Finally, our current design:

I still think this is a good looking site.

This design was also the first year we had a separate mobile site.

I’ve had the draft for this post written for years and I think it’s time just to hit publish on it – feel free to comment about all the things I forgot. There should probably also be some disclaimer about me writing this on a Friday afternoon and I take no responsibility for misinformation or wrong dates. :)

Thanks to fligtar for having screenshots of all the old versions!

Tagged , , ,

Improving our process; Part 1: Identifying Priorities

One of our stress points is juggling too many projects. It’s easy to hear ideas pitched from excited people and agree that we should do them, but it’s hard to balance that with all the other commitments we’ve already made. At some point we have too many “top priority” projects and people are scrambling to keep up with just the bare minimum – forgoing any hope of actually getting the enhancements done.

With a large number of projects, developers begin working independently to finish them which increases feelings of isolation, contributes to poor communication (“I don’t want to interrupt you”), and delays code review until someone can context switch over to look at it. Basically we’ve hit a point where – on large enough projects – working separately is slowing down our delivery and lowering our team morale.

In addition to being spread too thinly, there is also the question of why are we working on the projects we’re working on. In theory, it’s because they support our company goals, but that can often get muddled in the heat of the moment and interpretations and definitions (and even memory) can affect what we actually work on.

The work week helped highlight these two concerns and after several discussions (thanks to everyone who participated) we came up with some solutions. I’ll address them in reverse order – firstly, to determine priorities (the why), we wanted a more objective and public way of ranking the importance of projects. A public list of projects not only lines up with our open philosophy, but helps developers and the community know who is working on what, and also helps anyone requesting features get a feeling for a delivery timeline. Our solution was to build a grid with 16 relevant fields which we rank from 0-5 and then sum to get a final score. The fields are categorized and are broken down as follows:

  • Who finds value from the request? Separate points for consumers, developers, operators, OEMs, and Mozilla employees.
  • How does the request align with the company goals? Separate points for each goal.
  • How does the request align with the 2014 Marketplace goals? Separate points for the three goals.
  • Lastly, is there internal necessity? Separate points for Technological Advancement, Legal Compliance, and Urgency.

Once we have a request, a champion for the request meets with representatives from the product, UX, and developer teams, as well as any other relevant folks. We discuss the request and the value in the specific areas and then add up the score to get a number representative of priority. Clearly this isn’t a hard and fast science, but it’s a lot more objective than a gut feeling. It lets us justify decisions with reasoning, helps us think of all affected groups, and makes sure we’re working on what is important. If something has a high score we’ll start talking about timelines and moving things around, keeping in mind commitments we’ve made, planning and research that needs done, and any synchronizing we need to do with other dependent teams.

The other issue – people trying to do too much – is easily solved in a conversation but will likely be a challenge in practice. The plan is to cap our active tasks to 3 or 4 at any one time. Anything else will queue up behind them or displace them as appropriate. The exact number may change based on size of the team and size of the project, but my goal is to get 2 to 4 people working on a project together and with our current numbers that’s about what we can do.

I’m working on a real-time public view of our tasks but I’m traveling this week and haven’t had a chance to finish it. I’ll post again with a link once it is finished.

Update Feb 13: Added links to goals.

Tagged , , ,

Splitting AMO and the Marketplace

Years ago someone asked me what the fastest way to stand up an App Marketplace was. After considering that we already had several Add-on Types in AMO I replied that it would be to create another Add-on Type for apps, use the AMO infrastructure as a foundation for logins/reviews/etc. and do whatever minor visual tweaks were needed. This was a pretty quick solution but the plan evolved and “minor visual tweaks” turned into “major visual changes” and soon a completely different interface. Fast forward a few years and we have two separate sites (addons.mozilla.org and marketplace.firefox.com) running out of the same code repository but with different code. Much of the code is crudely separated (apps/ vs mkt/), but there are also many shared files, libraries, and utilities, both front and backend. The two sites run on the same servers but employ separate settings files.

There has been talk about combining the two sites so that the Firefox Marketplace was the one stop shop for all our apps/add-ons/themes/etc. but there was reluctance to move down that path due to the different user expectations and interfaces – for example, getting an app for your phone is a lot different flow than putting a theme on Firefox. While the debate has simmered with no great options the consequences of inaction continue to grow. Today’s world:

  • Automated unit tests which take far longer than necessary to run because they run for both sites.
  • Frustration for developers who change code in one area and affect a completely different site.
  • Confusion for any new employees or contributors as they struggle to set up the site (despite our decent documentation).
  • A one-size-fits-all infrastructure approach with no ability to optimize for the very different sites (API/service based vs standard MVC)

The best way to relieve the stress points above is complete separation of addons.mozilla.org and marketplace.firefox.com. Read the full (evolving) proposal. Feedback welcomed.

Tagged , , , , ,

Some new Marketplace dashboards

Cross posting from a mailing list:

I'm pleased to present a couple of dashboards I made over the break for the
Marketplace.  Behold:

1) How many bugs were closed this week?
   https://metaplace.paas.allizom.org/bugskiosk/

   I actually had the code for that one already laying around, so that was just
   to see if Andy was taking pull requests on metaplace. :) 

2) What waffles are what?
   https://metaplace.paas.allizom.org/waffles/

   A long standing complaint is that it's hard to tell what waffles we have on
   which site.  Not anymore - now you can instantly get two scoops of knowledge
   as a healthy part of this complete dashboard. (prod's API isn't live so
   nothing in that column yet)

3) That API seems slow, but maybe it's just me...
   https://metaplace.paas.allizom.org/apikiosk/

   A close runner up in concerns, some of our API speeds have been questionable
   but it's hard to point to a problem when we're trying to find a graphite
   graph (behind auth) or it's only happening to you and you can screenshot it
   but it may just be slow that day or something.  Now we have our own personal
   arewefastyet.com so we can point to real(-time) data on our API speeds
   (measured from around the world via pingdom).  See the other thread on this
   list ("[RFC] API performance standards") for more thoughts on this

Happy 2014!
Tagged , ,

Marketplace in Q4, 2013

Another set of slides summarizing Marketplace accomplishments in Q4 of 2013 (html). Behold the animated gifs!

Also, hat-tip to Will Kahn-Greene’s summary of Input in 2013.

2013 Q3 Accomplishments

The end of September was the same as every quarter lately – a whirlwind of summaries and demonstrations. I gave several presentations on what the Marketplace, SUMO, and Input teams did but I didn’t post anything on this [neglected] blog. It’s my party and I like to emphasize demos and screenshots over text so I’m going to inline the slides here (and hope they are interesting). A downloadable version is available (7.3M, pdf). The list of accomplishments for the quarter is both incomplete and collected from many people – my contribution here is simply doing some management work and putting together the presentation.



















A big thanks to the great folks working at Mozilla.

Tagged , , ,

Small teams and the Marketplace workflow

Towards the end of last year my routine was to find a weeks worth of bugs for each Marketplace developer, assign what I thought they could do in a week, and then meet with the person individually to talk about them and make sure we both thought it was a reasonable amount and the priorities were right. This worked fine for smaller teams, however, it’s not unusual for us to close 70+ bugs a week these days and it simply used too much time. I’d often end up skipping the folks who were comfortable grabbing bugs on their own, which was fine, but it didn’t encourage discussion within the team of who was working on what and kind of pushed people into working in isolation.

The last time the whole team was in the same room together (six months ago, now?) I proposed a new plan which we’ve been following ever since. The plan recognizes that the components within Bugzilla already split the Marketplace along logical lines (eg. API, Statistics, Reviewer Tools, etc.). Using those components as a template I asked everyone what areas they were most interested in and organized small groups which were responsible for each component. These groups meet once a week to do three things for each component:

  1. Review any new bugs in the component since last time and set priorities.
  2. Make sure the bugs people are working on are the most appropriate and that they are in the current milestone.
  3. If anyone is blocked on anything either fix it or escalate it.

There are generally only a few new bugs in each component each week so the meetings are often only a few minutes. These meetings make discussing current features a part of the natural development flow instead of making developers go out of their way to find out what is happening. If anyone is stuck or confused it’s easy to speak up and get help and it also gives anyone who is interested in that part of the Marketplace a place to go to ask questions (meetings are open to anyone).

The main concept behind the plan is that developers are aware of what our goals and commitments are and know better than I do what needs to be done to get there. If I’m in between them and closing bugs I’m just in the way. This plan empowers developers to be owners of their components and features and more officially shares the experience of triaging, prioritizing, and breaking specs apart into bugs.

I still go to as many of the component meetings as I can and help push things in the right direction but I have confidence that if I have a conflicting meeting or am focused on something else I can skip the meeting and there won’t be any critical bugs that don’t get addressed. Using the system has allowed us to add several more people to the team without having additional layers of management and still allows me to focus on coordinating with other teams at Mozilla instead of spending all my time on just the Marketplace team. Additionally, I still look at every bug that is filed for the Marketplace, but I’m routinely a couple days behind on generic bugmail. If a blocker bug is filed it is usually caught right away with multiple people watching their components.

The groups aren’t set in stone and several people have already switched around. They are simply a starting point – if nothing else, a bucket of bugs developers can go to when they have a couple hours to fill and they don’t want to spend their time sorting through our entire list of open bugs.

After several months of using the new system I’m happy to say that it has taken great strides towards solving the two most common concerns the marketplace team had: What are my teammates working on, and what am I working on next week/month? Our strategies will continue to evolve as we grow and change but this has been an effective change for where we are at right now.

Tagged , , ,

The sunset of getpersonas.com

Over four years ago getpersonas.com was built as a gallery to hold Personas – easy to make and use themes for Firefox. A lot of programs were supporting skins at the time, but I don’t know of any which were so simple (literally, move your mouse over an image on a webpage and it’s applied). A community of artists formed and many designs emerged – some amazing and popular, some focused on specific things (hat tip to CollieSmile), many for sports, or brands, or just people who wanted pictures of their family in a spot they noticed all day long.

Honestly, I thought they seemed gimmicky at first, but I started using them casually and they can be pretty fun. Every once in a while I’ll look up at my search box and see this guy and crack a smile and that’s enough reason to keep using them for me.

Now I’m treading down the dangerous path of naming names – a road full of mine shafts and sharp turns – and if I make a mistake, my apologies, please let me know or leave a comment. I wasn’t involved at the start of the Personas project but I think credit for the idea goes to Chris Beard, original add-on development to Myk Melez, original Firefox development work to the Mozilla Labs team, Toby Elliott and Ryan Doherty as the primary web developers, Stephen Donner and Krupa Raj for QA, and Jeremy Orem for IT Support. I know other teams worked on code here and there, but I think those were the primary contributors. I was focused on the engineering aspects so I’m not sure who to credit for the operational side of things – I know someone was managing day to day life at the time (it was Deb Richardson at some point, and today it’s Amy Tsay) and there were numerous volunteers reviewing submissions as they came in. It’s probably worth noting that in all the queues of all the sites at Mozilla the getpersonas.com queue was the only one I regularly saw completely finished and empty – kudos to those reviewers.

After seeing the rapid success of Personas we began talking about what we should do with them. The site didn’t have anyone adding features or doing maintenance on it – Ryan was the closest person to it and officially he was working on AMO. So, after some discussion we decided we would migrate all the data in getpersonas.com to AMO which would keep all of our themes and add-ons in one place. We finished the first steps (relatively) quickly – we wrote a questionable script which ran periodically and synced Personas listings from getpersonas.com to AMO. The files themselves and all the users remained on getpersonas.com. Looking purely at the number of times the script finished vs failed, it did pretty well, but there were times when it would fail constantly until someone could find time to fix it, much to the concern of the artists who didn’t see new data showing up on AMO.

Meanwhile, our security team decided to add getpersonas.com to the bug bounty program which surprised everyone and which we’ve playfully chided them about ever since. Some of our oldest and hacked together code (remember, this was originally an experiment) was now coming under scrutiny from security professionals and bored/gifted students from around the world. Special thanks to Matthew Noorenberghe and Brandon Savage for writing patches to fight back the flood of incoming security bugs.

A final spur to getting the migration finished came in the form of Mozilla Persona – an aptly named identity system, also from our company. After some spirited debate it was decided the identity project would use the Persona name and the personas on getpersonas.com would be referred to as Themes (and the existing themes we had as Complete Themes). With all the logistics out of the way we rapidly adopted the new names everywhere save getpersonas.com which continued to cling to life.

Late last year Chris Van Wiemeersch spearheaded a clandestine operation to finally finish closing down getpersonas.com (with special mention to Kevin Ngo for contributing piles of code all the while finishing his final exams, and Kris Maglione for updating the Personas Plus add-on).

After another quarter of late nights balancing the migration with developing the Firefox Marketplace it’s finally happened: I’m happy to announce that getpersonas.com has now officially retired and all the content and artist accounts have been migrated to AMO.

It’s exciting from a Mozilla point of view because the confusion of projects can finally end and we can stop pretending to maintain a site that we haven’t actually done anything with for years. The artists and users have reason to be happy also, though – moving to AMO means it’s on an actively maintained site. Improvements can happen, features can be added, and new ideas can be explored. Finishing this long awaited migration will breathe new life into the themes community and I’m looking forward to all the fun new ways we’ll see Perso-uhh – I mean – themes, being used.

A screenshot of new users on AMO on the night we imported the accounts. Our usual numbers of 1500/day are eclipsed by 1.5 million that night. Also a testament to the site’s scalability that it didn’t miss a beat.

Tagged , , ,

Inception: projects within projects

When we converted addons.mozilla.org from PHP to Python I mentioned how deceptively large (lines of code) the site had grown with so many views and features. We’ve since built marketplace.firefox.com on literally the same code base (a decision I hoped to blog about some day, alas) and the line count continues upwards. The MVC paradigm is one of the most common out there, easy to understand, and built in to our frameworks so it was naturally a good choice. However, as the site continues to expand we’ve had to review what the best architecture for the code would be and the main point that stands out is that our entire site and its services are a monolithic blob of code. Both sites, all of our APIs, all of the PayPal code, the administrative tools, the developer control panel, the reviewer queues, the list goes on – it’s all in one repository and changing one means potentially affecting all of them.

With big plans for 2013 and no signs of slowing our code expansion we’ve hit the point where it makes more sense to break the site up into smaller projects. We’ve already started with our payment processor and metrics processor and more are soon to follow. These projects will be tied together around a central API on the back end. We’ve had partial front end APIs for years (like search) which we will expand to make the entire site accessible via API. At that point our front end code will be completely separated and can be optimized to speak directly to JSON APIs instead of pulling HTML with every request (something which the developers have been demoing the past couple weeks at our show and tells). A rough diagram is below, subject to change, of course.

Marketplace Architecture Diagram

Some benefits of this approach include:

  • Physical separation of code which will easily allow us separate SLAs or even separate conformance agreements, like PCI compliance
  • Speaking to APIs means we’ll have specific failure cases we can write code for so when a piece of the site dies the entire site doesn’t crash
  • Each project can be written to scale horizontally without affecting the other projects
  • Developers and contributors can focus on a single project without having to check out the entire site (and deal with substantial configuration maintenance)
  • Localizations will be broken up into separate .po files, a long requested feature from our localizing friends
  • Deployments can happen for specific projects without updating all of them. These will be faster, easier to scope, and easier to roll back if necessary
  • 3rd parties can build full featured sites off our APIs

There are a couple of downsides: maintenance/configuration of each project and integration testing. The former is diminished a bit because not every developer works on every project so with some prudent mocking and a few virtual machines there should be an easy flow there. The second is something we currently neglect but will become critical with these changes so expect some solid developments there. With all the other work on our schedule I think we’ll be touching parts of this project in every quarter this year. If you hear the codename INCEPTION it’s referring to this project.

Tagged , , , , ,