Blog

On Friday, January 28th I hopped in a cab and went up to NPR’s Bryant Park recording studio for a fifteen minute chat with Ira Flatow, host of Science Friday. I’ve been a big fan of Ira and Science Friday since I discovered the show years ago, and it was a very exciting honor to be a guest.

The image at the right is a snapshot I took with my phone while nervously waiting outside the studio.

The title of the segment is the rather dramatic Privacy At Stake As Sites Track Online Preferences. Our conversation wound around the issues of tracking user data online, and the potential opportunities and dangers that all users of online services face.

NPR has the full broadcast and transcript online.

By far the most fun and unexpected aspect of this was the number of people who wrote to me to ask questions or say that they appreciated my perspective. Many of them don’t typically follow technology news or startups, and it’s exciting to hear from people who heard the interview and were intrigued.

The video from my keynote at Strange Loop 2010 is up!

You can watch the video here: Machine Learning: A Love Story

The original abstract:

Machine learning has come a long way in recent years — from a long-marginalized field so old it still has the word “machine” in the name, to the last, best hope for making sense of our massive flows of data.

The art of ‘data science’ is asking the right questions; the answers are generally trivial or impossible. This talk will focus more on questions than on answers. I’ll give a brief history of the field with a focus on the fundamental math and algorithmic tools that we use to address these kinds of problems, then walk through several descriptive and predictive scenarios.

Finally, I’ll show one example system using bit.ly data in-depth, from the backend infrastructure through the algorithms and data processing layer to show a functioning product.

Attendees should expect to hear some good stories of data gone right and data gone awry, and walk away with a few new clever tricks.

The presentation was calibrated for the audience in the room, but I’ll be happy to answer any questions in the comments below!

How can twitter be so popular and successful if it’s down all the time?

We base statements like this on the assumption that quality of a web application maps linearly to the application’s stability. This is obviously true for most sites most of the time, but things get interesting at the edge where rare, unpredictable failure actually enables more complex human interactions around the service.

Unlike e-mail, twitter etiquette doesn’t demand that you read or reply to every message from every person you follow (or who follows you). Combine that lightweight social touch with occasional technical issues and human communication patterns, and we start to see some interesting behavior.

Twitter’s lack of reliability as a platform allows us to use the technical failings to mask our own social imperfections. How often have you heard or said something like “I was sure I was following you” or “I must not have gotten that DM” or even “I think I tweeted that…”? Even just a small percent of users behaving this way changes the social expectations.

I’d love to construct an experiment to figure out whether this idea has merit, and if so, what the optimal amount of unavailable operations for social deniability is. Should 1 in 100 actions fail? 1 in 10,000? 1 in 1,000,000? Does it matter if any fail, as long as we believe that every so often failure occurs? (How often do things really get lost in the mail, anyway?)

It’s amusing to conceive of a system that succeeds socially because it often fails technically.

I received this e-mail via my contact form:

I just discovered you via a Google search because I’m highly considering attending this year’s upcoming Hadoop World in NYC. I appreciate your page that you wrote up after attending last year’s event. I’m wondering if you feel that Hadoop has enough momentum and support to be a “here to stay” technology worth investing one’s time and education into, or is it possible it might fade and be deprecated by something else as the need for big data analysis continues to grow? …

I’ve had a few similar conversation with people lately, and I thought posting my response might help others making similar decisions. The e-mail is referencing my post from last year’s hadoop world NYC.

Thanks for reaching out. There are several questions in your message
and I hope that this will address them all.

This IS an extremely exciting time to be alive and working with data.
We now have the capacity to learn thing about our systems, people in
general, and the world that we simply couldn’t know before — the
field is only going to keep growing.

Hadoop is currently the primary tool framework for this kind of data
analysis. It’s certainly worth learning now, especially since Amazon’s
elastic mapreduce makes it very easy for individuals and small team to
get started without a large investment.

I’m also not a huge fan of Java and I wish more resources were going
into non-Java alternatives. Fortunately, you can use hadoop via the
streaming API in most any language you choose.

I do think it’s important to separate the discussion of tools from the
larger philosophical discussion of open problems, algorithms, and
techniques. You can learn the tools from books and blogs. The real
reason to go to a conference like Hadoop World is to meet the people
who go to conferences like Hadoop World and get into those deeper
conversations.

I do hope this year’s conference will highlight the difference between tools and methods, and will also provide plenty of space for those casual hallways conversations.

Several months ago, on a whim inspired by an off-hand comment from Chris, I created a bot to bring the wonders of the Unix bc language to twitter.

bc is a command-line calculator that’s fast and has the capacity to do some fairly complex math.

Try it out on the command line:

echo '100 / 10' | bc -l

…Or by sending a direct message to bc_l (if you follow bc_l it will follow you back within a few hours).

I released the code under GPL, and it’s available on github: http://github.com/hmason/tweetbc.

John Cook mentions the bot and makes some great observations in his post three surprises with bc.