Test Pilot Legacy Program Final Review

One of my Q3 goals was to migrate the Legacy Test Pilot users into our new Test Pilot program (some background on the two programs). The previous program was similar in that people could give feedback on experiments, but different enough that we didn't feel comfortable simply moving the users to the new program without some kind of notification and opting-in.

We decided the best way to do that was simply push out a new version of the legacy add-on which opened a new tab to the Test Pilot website and then uninstalled itself. This lets people interested in testing experiments know about the new program without being overbearing. Worst case scenario, they close the tab and have one less add-on loading every time Firefox is started.

In our planning meeting it was suggested that getting three percent of users from the old program to the new would be a reasonable compromise between realistic and optimistic. I guffawed, suggested that the audience had already opted-in once, and put 6% in as our goal and figured it would be way higher. Spoiler alert: I was wrong.

I'll spare you the pain of writing the add-on (most of the trouble was that the legacy add-on was so old you had to restart Firefox to uninstall it which really broke up the flow). On August 24th, we pushed the update to the old program.

In the daily usage graph, you can see we successfully uninstalled ourselves from several hundred thousand profiles, but we still have a long tail that doesn't seem to be uninstalling. Fortunately, AMO has really great statistics dashboards (these stats are public, by the way) and we can dig a little deeper. So, as of today there are around 150k profiles with the old add-on still installed. About half of those are reporting running the latest version (the one that uninstalls itself) and about half are disabled by the user. I suspect those halves overlap and account for 75k of the installed profiles.

The second 75k profiles are on older add-on versions and are not upgrading to a new version. There could be many reasons when we're dealing with profiles this old: they could be broken, they might not have write permissions to their profile, their network traffic could be being blocked, an internet security suite could be misbehaving, etc. I don't think there is much more we can do for these folks right now, unfortunately.

Let's talk about the overall goal though - how many people joined the new program as a result of the new tab?

As of the end of Q3, we had just over 26k conversions making for a 3.6% conversion rate. Quite close to what was suggested in the original meeting by the people who do this stuff for a living, and quite short of my brash guess.

Overall we got a 0.6 score on the quarterly OKR.


Since I'm writing this post a few weeks after the end of Q3, I can see that we're continuing to get about 80 new users per day from the add-on prompt. Overall that makes for about 28.5k total conversions as of Oct 27th.

Test Pilot 2016 Q4 OKRs

The Test Pilot 2016 Q4 OKRs are published. Primarily we'll be focused on continued growth of users (our overall 2016 goal). We deprioritized localization last quarter and over-rotated on publishing experiments by launching four when we were only aiming for one. This quarter we'll turn that knob back down (we're aiming for two new experiments) and get localization done.

We also failed to graduate any experiments last quarter -- arguably the most important part of our entire process since it includes drawing conclusions and publishing our results. This quarter we'll graduate three experiments from Test Pilot, publish our findings so we can improve Firefox, and clear out space in Test Pilot for the next big ideas.

testpilot.firefox.com just got a lot easier to work on

We originally built Test Pilot on top of Django and some JS libraries to fulfill our product requirements as well as keep us flexible enough to evolve quickly since we were a brand new site.

As the site has grown, we've dropped a few requirements, and realized that we were using APIs from our engagement team to collect newsletter sign ups, APIs from our measurement team for our metrics, and everything else on the site was essentially HTML and JS. We used the Django scaffolding for updating the experiments, but there was no reason we needed to.

I'm happy to highlight that as of today testpilot.firefox.com is served 100% statically. Moving to flat files means:

  • Easier to deploy. All we do is copy files to an S3 bucket. No more SQL migrations or strange half-pushed states.

  • More secure. With just flat files we have way less surface area to attack.

  • Easier to participate in. You'll no longer need to set up Docker or a database. Just check out the files, run npm install and you're done. (disclaimer: we just pushed this today, so we actually still need to update the documentation)

  • Excellent change control. Instead of using an admin panel on the site, we now use GitHub to manage our static content. This means all changes are tracked for free, we already have a process in place for reviewing pull requests, and it's easy to roll back or manipulate the data because it's all in the repository already.

If you want to get involved with Test Pilot, come join us in #testpilot (or webchat)!

Test Pilot Q3 OKR Review

For the third quarter of 2016 the Test Pilot team decided to try using the OKR method (an OKR overview) for our goal setting.

We all sat down in London and hashed out what direction we wanted to move in for Q3, what we thought we could do in that timeframe, prioritized the results and then I published the results on the wiki. If you're interested in what Test Pilot did in Q3 you should read that link because it has a bunch of comments in it.

I knew we deprioritized some of our goals mid-quarter, but I was surprised to see us come up with a pretty modest .61. My takeaways from my first time using the OKR method is:

  • Wording is really important. Even if you all agree on some words while sitting around a table, look them over again the next day because they might not make as much sense as you think.

  • Getting the goals for your quarter planned before the quarter starts is tops.

  • Having a public list of goals you can point people to is great for your team, other teams you work with, and anyone in the community interested in your project.

  • Estimates for how long things will take you is still a Really Hard Problem.

The feedback I've received about the OKR process we followed has been really positive and I expect to continue it in the future.

Test Pilot is launching three new experiments for Firefox

The Test Pilot team has been heads-down for months working on three new experiments for Firefox and you can get them all today!

Min Vid

Min Vid is an add-on that allows you to shrink a video into a small always-on-top frame in the corner of your browser. This lets you watch and interact with a video while browsing the web in other tabs. Opera and Safari are implementing similar features so this one might have some sticking power.

Thanks to Dave, Jen, and Jared for taking this from some prototype code to in front of Firefox users in six months.

Tracking Protection

Luke has been working hard on Tracking Protection - an experiment focused on collecting feedback from users about which sites break when Firefox blocks the trackers from loading. As we collect data from everyday users we can make decisions about how best to block what people don't want and still show them what they do. Eventually this could lead to us protecting all Firefox users with Tracking Protection by default.

Page Shot

Page Shot is a snappy experiment that enables users to quickly take screenshots in their browser and share them on the internet. There are a few companies in this space already, but their products always felt too heavy to me, or they ignored privacy, or some simply didn't even work (this was on Linux). Page Shot is light and quick and works great everywhere.

As a bonus, a feature I haven't seen anywhere else, Page Shot also offers searching the text within the images themselves. So if you take a screenshot of a pizza recipe and later search for "mozzarella" it will find the recipe.

I was late to the Page Shot party and my involvement is just standing on the shoulders of giants at this point: by the time I was involved the final touches were already being put on. A big thanks to Ian and Donovan for bringing this project to life.

I called out the engineers who have been working to bring their creations to life, but of course there are so many teams who were critical to today's launches. A big thank you to the people who have been working tirelessly and congratulations on launching your products! :)