Blog

Posted: April 1, 2013 | Author: Hilary Mason | Filed under: blog | Tags: bitly, data, engineering, infrastructure | 3 Comments »

Data engineering is when the architecture of your system is dependent on characteristics of the data flowing through that system.

It requires a different kind of engineering process than typical systems engineering, because you have to do some work upfront to understand the nature of the data before you can effectively begin to design the infrastructure. Most data engineering systems also transform the data as they process it.

Developing these types of systems requires an initial research phase, where you do the necessary work to understand the characteristics of the data, before you design the system (and perhaps even requiring an active experimental process where you try multiple infrastructure options in the wild before making a final decision). I’ve seen numerous people run straight into walls when they ignore this research requirement.

Forget Table is one example of a data engineering project from our work at bitly. It’s a database for storing non-stationary categorical distributions. We often see streams of data and want to understand what the distributions in that data look like, knowing that they drift over time. Forget Table is  designed precisely for this use, allowing you to configure the rate of change in your particular dataset (check it out on github).

Posted: March 23, 2013 | Author: Hilary Mason | Filed under: blog | Tags: google | 8 Comments »

Google Now is an extension to Google’s Android search app that uses all of the data that Google has about you along with what it can guess about your current context to present the information it thinks you need when it thinks you need it.

It’ll tell you to leave a bit early to make your next calendar event because of heavy traffic, or that it’s a friend’s birthday, or that there’s a cool cafe nearby where you are.

I think it’s amazing.

It’s amazing because this is the first Google product that takes ALL OF THE DATA that they have about us and actually makes it useful for us. Not for advertisers.

Finally.

Posted: February 25, 2013 | Author: Hilary Mason | Filed under: blog | Tags: blogs, data science | 4 Comments »

I’m gathering a bundle of data science blogs to share. I’m looking to include blogs that update regularly and aren’t either personal opinion and project blogs (like this one) or primarily about marketing any particular company. Let me know if you have a favorite that I’ve forgotten.

If you’re just looking for one place to start, hop on over to Simply Statistics.

Posted: February 4, 2013 | Author: Hilary Mason | Filed under: blog | 4 Comments »

I ended up at NYC Resistor on Sunday, and decided to experiment with physical visualization of some data. I grabbed the clicks per second on keyphrases including my name (“hilary mason”) over the last six months, aggregated them by day, and made this graph:

This is easy enough to construct for any phrase using the clickrate data that we’re calculating at bitly. I exported it from matplotlib in svg, added a label, and used the laser-cutter to create this out of plywood:

laser-cut time series

…which will shortly be adorning my desk at work. This is very simple, but there’s a lot of fun to be had with the physical manifestation of patterns we see in large amount of ephemeral data.

Posted: January 29, 2013 | Author: Hilary Mason | Filed under: blog | 3 Comments »

Hilary Mason, Bing Celebrity

I have a Google alert set up for my name, and over the weekend it sent me here.

Update: Bing has removed the page and now redirects to a regular search.

It’s a page on Bing Celebrities, merging my information with information about Hilary Mason, the (now deceased) British actress. According to this page, I have starred in movies before I was born and made videos after I died. It’s my photo and her filmography.

It’s creepy, but it’s also intriguing. How does this happen?

The data is credited to AMG and inbaseline, whose domain, though linked directly from Bing, does not resolve. Entity disambiguation is certainly a challenge, but I expect more from Microsoft, with so much data and so many brains.

This kind of error makes it extremely clear that identity is not a solved problem. I’ve written a bit about identity slippage before. And that people are especially sensitive to errors about themselves.

This isn’t the first time a search engine has confused me with the other Hilary Mason, except the first time was cuil (remember that?) and it was her photo as Ugly Hag and my bio. I’ll take it Bing’s way, thank you!