The framework was written in an extensible way so in addition to tweaking the built-in searches, we could also leverage external scripts. The first such script that is making it to the live site is Adrian Kalla’s localization completeness check. This script attempts to parse and record all the English string files as a baseline. Then it looks at each locale and reports any missing files, missing translations, or untranslated strings (translations that exist in the locale but are the same as English).

If you validate an extension now and only have partial L10n coverage, scroll down to the new L10n section and you should see something like this:

Thanks to RJ and Adrian for doing all the work on this.

During the week of April 29, 2009 and May 5, 2009 there were around 150000 queries. Of the top 20 queries on addons.mozilla.org (a quick estimate says that is around 12% of the total queries on the site) only 7 actually have search terms. The rest are just choosing different options for the search like category or number of results on a page. If we filter the top queries for ones that include search terms we get a graph that looks like this:

All the searches on that page are for the en-US locale unless otherwise noted. It looks like the majority of searches are for specific add-ons but there are also some popular generic terms like download, gmail, and video. I think it’s interesting that German was the only other locale to make the list (and fairly high up on the list). Maybe the next stats post will be about overall locale use.

Statistics for both update pings and download counts have been updated beginning with February 1 through today, February 6th. Some notes:

• New statistics are stored in UTC and data processing happens shortly after the logs close. This means you can expect new data at around 8pm PST or shortly after.
• Download numbers will drop dramatically. They have been recorded incorrectly[1] for the past several weeks. Bug 472538 has more details.
• We’ll begin replacing statistics back to 2008-11-15 over the next few weeks as processing time allows.
• An aside that you may not know: When Firefox looks for an update to an add-on we count that as an “update ping.” If it finds the update it will hit releases.mozilla.org directly for the new add-on. That means that in your current stats numbers updates are not counted as downloads, or another way, “download counts” are the counts of someone actually clicking the “Install Now” button on addons.mozilla.org.

Since we’re pulling these statistics from a team dedicated to crunching numbers we’re getting richer and more reliable data now. This frees up our time to fix existing stats bugs and also to add additional data views (like what locale your users are using). Good things are coming; keep an eye on your stats!

Update 2008-02-07: HP issued a critical alert regarding potential data loss which affected our servers. Our IT team applied the fix but upon restart discovered it’s been way too long since the file system had fsck run on it. Since there is so much data on the system it will take several more hours to finish, then IT will restore log files, and then we can begin to process the stats for this weekend. In short, stats won’t be current for another day or two.

[1] The technical reason is that Firefox does 2 or 3 GET requests to a server when it installs an add-on. The filter we had to remove duplicate requests was broken.

Our current process is to count download statistics once per day and update ping statistics once per week (update pings are a sampling of the complete set). The reliability of the script generating these statistics has been falling as our data size has grown and we’ve had several bugs filed regarding the numbers it’s produced. Most of the time they are relatively small fixes and the script continued to limp along.

Currently we’re facing questionable results in both sets of statistics (bug 468570 for update pings, bug 472538 for download counts). I’ve been debugging the update pings script and despite solving some problems we’re continuing to see the script fail to run properly.

Parallel to AMO development, Daniel Einspanjer has been working on a larger statistics parser that will aggregate data from many Mozilla sites into a dashboard with easy visualizations. It turns out he’s already processing the AMO logs and pulling out more data than us more often and in less time.

With a system like that available it doesn’t make sense for us to continue to develop (and, in this case heavily modify) our local statistics scripts. With that in mind, our next steps are:

1. Verify the results we (used to) get with the AMO scripts match those of the new system
2. Create a transformation script to push the data from Daniel’s project to the AMO database
3. Turn off the AMO scripts
4. Back fill statistics through at least November 15th, 2008 to replace our flailing stats. If the comparisons in step 1 reveal miscounting from before that we’ll back fill as far as we need to.

These steps will let us meet the immediate goal of getting the statistics we offer now to be reliable and complete. In the future we can look at pulling additional data from the new metrics system. The target date to switch to the new system is the end of next week, Jan 31 2009. Once we make the switch we can evaluate how long the parsing takes and give an estimate of how long back filling will take. As always, let me know if there are any concerns.

Update 2009-02-02: We compared the scripts’ results and found a discrepancy among add-ons that have significant external download numbers. The current stats script verified the GUID matches and then counted the update. The new stats script verified the GUID and the version before counting the update. This means if a specific version isn’t hosted on AMO the new script doesn’t count it. I think the current method of verifying only the GUID is more useful to authors and the new script is being changed. That means we’ll have to re-run and re-compare the numbers (a single day is taking about 5 hours now). Other numbers are showing early promise. I’ll continue to update as we progress.

A few straggling bugs aside[1] proper message sorting has been achieved and the ThreadBubble extension is no longer needed.

The latest version, ThreadBubble 0.8, is compatible with Thunderbird versions up to 3.0a2pre and I expect it will be the last version released. Maybe I’ll work on a Firefox extension next…

Anyway, thanks to everyone who tested, used, and gave feedback about ThreadBubble.

[1] This is kind of a joke – the parent bug is actually a meta bug for all threaded view issues and I don’t know how many of those are confirmed or are relevant to what ThreadBubble fixed. I do know I filed bug 461100 last night which is a new problem with the threaded view as far as I can tell.

• Thunderbird 3 Support: Tested on the latest nightly and working fine. This also means a lot of the code was cleaned up and made simpler. (Thunderbird 2 still works)
• Fixed a bug when sorted ascending where new messages wouldn’t sink to the bottom of the list.

Download ThreadBubble 0.8

Someone already tested this update and had no problems so I’m putting it online. If I don’t hear any complaints by next week I’ll push it live on AMO and everyone can get the update automatically. Thanks!

Want to know the big changes and what’s brand new in Firefox 3? Check out Firefox 3 for developers.

Want to get right to the part about updating your extensions for Firefox 3? We’ve got a wiki page for that too.

If you know your extension will work and you just want to know what you need to do on AMO check out Justin’s quick review.

For more personalized assistance feel free to ask your question in #extdev on irc.mozilla.org.

If you’ll be in Brussels, Belgium at the end of this month (February 23rd and 24th) you should consider hitting up Mozilla’s Extension Developer’s Workshop at FOSDEM 2008. It promises a trifecta of awesomeness including helpful people, Mozilla swag, and delicious beer.

Edit: update first link to be more useful

function someListener() { }

someListener.prototype =
{
  OnItemAdded: function(parentItem, item) {
      alert('You got mail!');
  }
}

var mailSession = Components.classes["@mozilla.org/messenger/services/session;1"].getService(Components.interfaces.nsIMsgMailSession);

mailSession.AddFolderListener(new someListener(),Components.interfaces.nsIFolderListener.added);

This code is more verbose than you need (declaring the listener function), and will get you more events than you need (you’ll have to filter the ones you want in your function).

After researching new methods of being notified for ThreadBubble, I came across a better method available in Thunderbird 2.0 – using the nsIMsgFolderNotificationService. If you look at the interface, you can see it distinguishes between an item being added, deleted, moved, etc. which makes it easy to pick out the events you want to watch.

The example above using the new interface looks very similar, but I think it comes out a lot cleaner:

var someListener = {
    itemAdded: function(item) {
      alert('You got mail!');
    }
}

var notificationService = Components.classes["@mozilla.org/messenger/msgnotificationservice;1"].getService(Components.interfaces.nsIMsgFolderNotificationService);

notificationService.addListener(someListener);

This method isn’t affected by the bug I mentioned earlier, and it’s a more direct method of getting notified.

The examples here are sparse on details – you should always be prepared to catch exceptions from code like this (put it in a try/catch block). You can see a complete example in the ThreadBubble source code.

Fixing this problem was more involved than I was hoping, and while debugging, I stumbled across what I’m considering a bug in Thunderbird:

If there is a new message in a thread, and I delete the root message, the OnItemAdded event is fired as if a new message just arrived. If I delete any message other than the root, or if there is no message marked “new” in the thread, the event is not fired. From my limited understanding of the system, it seems like OnItemAdded should only be fired when an item is added (either new mail arrives, or I move a message from another folder). Regardless, I’m no longer using the OnItemAdded hook, so it’s not an issue for ThreadBubble any more.

It’s also worth noting that ThreadBubble now only works with Thunderbird version 2.0 and above due to the way it gets notified about new mail.

Download ThreadBubble 0.5