Enabling Encryption in Weechat

Here are some step by step instructions for enabling encryption using crypt.py in weechat.

First, ensure the crypt.py script is installed. The easiest way is from within weechat itself:

/plugin load script
/script install crypt.py
/script autoload crypt.py

You should see some simple messages saying crypt.py was installed and enabled for automatic loading.

Next you need to generate an encryption key. This just needs to be named with the channel you want to use the key with (I use #channel below). For example:

cd .weechat
openssl genrsa -out cryptkey.#channel 4096

We might as well make sure only we can read it:

chmod 600 cryptkey.#channel

At this point typing any text in #channel will automatically be encrypted (use a second client if you'd like to verify it). For example, I typed:

1811 clouserw │ testo

And the other clients in #channel see:

1811 clouserw | +qRy3GsV2sPRlJSdP1IqqV|

The next step is to distribute the key to the other people who will need to decrypt the chat. Take a minute to consider the best way to do this as the chat will only be as secure as this key.

Lastly, you can optionally add an indicator to the status bar by adding 'encrypted' to weechat.bar.status.items. This command will tell you your current value:

/set weechat.bar.status.items

Copy that value and add 'encrypted' where you'd like it to show up. Mine is:

/set weechat.bar.status.items "[time],[buffer_count],[buffer_plugin],buffer_number+:+buffer_name+{buffer_nicklist_count}+buffer_filter,encryption,[lag],[hotlist],completion,scroll"

which looks like this:

[18:16] [32] [irc/freenode] 30:#channel{3}⚑ (encrypted)  [H: 3, 4]

That's all there is too it!

Marketplace and Payments Systems Diagrams

A couple years ago Krupa filled up a whiteboard with boxes and arrows, diagramming what the AMO systems looked like. There was recently interest in reviving that diagram and seeing what the Marketplace systems would look like in the same style so I sat down and drew the diagrams below, one for the Marketplace and one for Payments.



Honestly, I appreciate the view, but I wince at first glance because of all the duplication. It's supposed to be "services from the perspective of a single service." Meaning, if the box is red, anything connected to it is what that box talks to. Since the webheads talk to nearly everything it made sense to put them in the middle, and the dotted lines simply connect duplicate services. I'm unsure whether that's intuitive though, or if it would be easier to understand if I simply had a single node for each service and drew lines all over the diagram. I might try that next time, unless someone gives me a different idea. :)

Lastly, this is the diagram that came out first when I was trying to draw the two above. It breaks the Marketplace down into layers which I like because we emphasize being API driven frequently, but I'm not sure the significant vertical alignment is clear unless you're already familiar with the project. I think finding a way to use color here would be helpful - maybe as a background for each "column."

Or maybe I'm being too hard on the diagrams. What would you change? Are there other areas you'd like to see drawn out or maybe this same area but through a different lens?

Goodwill Updates - A Firefox OS Feature Idea

A common aspect amongst the regions Firefox OS targets is a lack of dependable bandwidth. Mobile data (if available) can be slow and expensive, wi-fi connections are rare, and in-home internet completely absent. With the lack of regular or affordable connectivity, it’s easy for people to ignore device and app updates and instead opt to focus on downloading their content.

In the current model, Firefox OS daily pings for system and app updates and downloads the updates when available. Once the update has been installed, the download is deleted from the device storage.

What if there was an alternative way to handle these numerous updates? Rather than delete the downloads, they are saved on the device. Instead of each Firefox OS device being required to download updates, the updates could be shared with other Firefox OS devices. This Goodwill Update would make it easier for people to get new features and important security fixes without having to rely on internet connectivity.

a concept drawing

Goodwill Update could either run in the background (assuming there is disk space and battery life) or could be more user-facing presenting people with notifications about the presence of updates or even showing how much money has been saved by avoiding bandwidth charges. Perhaps they could even offer to buy Bob a beer!

Would this be worth doing to help emerging markets stay up to date?

PS. Hat tip to Katie and Tiffanie for the image and idea help.

Altering large tables without bringing down your service

When we run ALTER statements on our big tables we have to plan ahead to keep from breaking whatever service is using the database. In MySQL, many times* a simple change to a column (say, from being a short varchar to being a text field) can read-lock the entire table for however long it takes to make the change. If you have a service using the table when you begin the query you'll start eating into your downtime budget.

If you have a large enough site to have database slaves you'll have a double-whammy - all reads will block on the master altering the table, and then, by default, the change will be replicated out to your slaves and not only will they read-lock the table while they alter it, but they will pause any further replication until the change is done, potentially adding many more hours of outdated data being returned to your service as the replication catches up.

The good news is, in some situations, we can take advantage of having database slaves to keep the site at 100% uptime while we make time consuming changes to the table structure. The notes below assume a single master with multiple independent slaves (meaning, the slaves aren't replicating to each other).

Firstly, it should go without saying, but the client application needs to gracefully handle both the existing structure and the anticipated structure.

When you're ready to begin, pull a slave out of rotation and run your alter statement on it. When it completes, put the slave back into the cluster and let it catch up on replication. Repeat those steps for each slave. Then failover one of the slaves as a new master and pull the old master out of rotation and run the alter statement on it. Once it has finished put it back in the cluster as a slave. When the replication catches up you can promote it back to the master and switch the temporary master back to a slave.

At this point you should have the modified table structure everywhere and be back to your original cluster configuration.

Special thanks to Sheeri who explained how to do all the above and saved us from temporarily incapacitating our service.

*What changes will lock a table vary depending on the version of MySQL. Look for "Allows concurrent DML?" in the table on this manual page.

Contributing a patch to the Firefox Marketplace from scratch

Jared, Stuart, and Andy recently spent some time focusing on one of the Marketplace's biggest hurdles for new contributors: how do I get all these moving pieces set up and talking to each other?

I haven't written a patch for the Marketplace in a while so I decided to see what all the fuss was about. First up I, of course, read the installation documentation. Ok, I skimmed it, but it looks pretty straight forward.

Step 1: Install Docker

I'm running Ubuntu so that's as easy as:

sudo apt-get install docker.io

To fix permissions (put your own username instead of clouserw):

sudo usermod -a -G docker clouserw

Step 2: Build Dependencies

The steps below had lots of output which I'll skip pasting here, but there were no errors and it only took a few minutes to run.

$ git clone https://github.com/mozilla/wharfie
$ cd wharfie

$ bin/mkt whoami clouserw
$ bin/mkt checkout

$ mkvirtualenv wharfie
$ pip install --upgrade pip
$ pip install -r requirements.txt
$ sudo sh -c "echo  mp.dev >> /etc/hosts"

Step 3: Build and run the Docker containers

I ran this seemingly innocent command:

$ fig build

And 9 minutes and a zillion pages of output later I saw a promising message saying it had successfully built. One more command:

$ fig up

and I loaded http://mp.dev/ in my browser:

Screenshot of Marketplace

A weird empty home page, but it's a running version of the Marketplace on my local computer! Success! Although, I'm not sure it counts unless the unit tests pass. Let's see...

$ CTRL-C  # To shutdown the running fig instance
$ fig run --rm zamboni python ./manage.py test --noinput -s --logging-clear-handlers
Ran 4223 tests in 955.328s
FAILED (SKIP=26, errors=34, failures=17)

Hmm...that doesn't seem good. Apparently there is some work left to get the tests to pass. I'll file bug 1082183 and keep moving. I know Travis-CI will automatically run all the tests on any pull request so any changes I make will still be tested -- depending on the changes you make this might be enough.

Step 4: Let's write some code

If I were new to the Marketplace I'd look at the Contributing docs and follow the links there to find a bug to work on. However, I know Bug 989121 - Upgrade django-csp has been assigned to me for six months so I'm going to do that.

I'll avoid talking about the patch since I'm trying to focus on the how and not the what in this post. The code is all in the /trees/ subdirectory under wharfie, so I'll go there to write my code. A summary of the commands:

$ cd trees/zamboni
$ vi <files>  # Be sure to include unit tests
$ git add <files>
$ git checkout -b 989121  # I name my branches after my bug numbers
$ git commit  # commit messages must include the bug number
$ git push origin 989121

Now my changes are on Github! When I load the repository I committed to in my browser I see a big green button at the top asking if I want to make a pull request.

Github pull request button

I click the button and submit my pull request which notifies the Marketplace developers that I'd like to merge the changes in. It will also trigger the unit tests and notify me via email if any of them fail. Assuming everything passes then I'm all done.

This flow is still a little rough around the edges, but for an adventurous contributor it's certainly possible. It looks like Bug 1011198 - Get a turn-key marketplace dev-env up and running is tracking progress on making this easier so if you're interested in contributing feel free to follow along and jump in when you're comfortable.