Blog

I recently attended the Third Annual Workshop on Search and Social Media, an academic workshop with very strong industry participation. The workshop was packed, and had some of the most informative and interesting panel discussions I’ve seen (not counting the one I spoke on!).

Daniel Tunkelang did a great job of writing up the specific presentations on his site and on the ACM blog, so I won’t attempt to re-create the presentations line by line at this late date. Rather, I’d like to highlight a few open problems and research questions that came out of the discussions that I hope to see developed in the next year.

Social search consists of a set of problems including (but hardly limited to) search of social content like status updates, real-time search, generating, labeling, and finding user-generated content, ‘long-tail’ events and interests, finding vs re-finding, and trend identification.

What data is available to social search? There are many kinds of social data, from e-mail (private) to blogs (public) and tweets (mostly public) — what is and should be searchable? How do we handle issues of privacy and identity management?

How do we compute relevance, taking into account freshness, accuracy, and degrees of social separation?

Will the architecture of these search engines look like the search engines we’re currently familiar with?

How do we evaluate accuracy and truthiness of social data?

How do we characterize social connections, around concepts like strong vs weak ties, and friend-of-a-friend vs friend-of-a-friend’s-friend? Can we converge on a single social graph representation?

How do we best filter social data to lead to accurate recommendations for content discovery? How do we accommodate the fact that as we move beyond static factual data, two people using the same query may be looking for very different results?

Finally, how do we deal with the chasm between the industry participants (who have LOTS of data) and the academic participants, who suffer from a lack of public (and publishable) data?

Thanks again to the organizers – Eugene Agichtein, Marti Hearst, Ian Soboroff, and Daniel Tunkelang – who put together a fantastic event.

For more on this and a cool demo, check out Gene Golovchinsky’s look at the SSM2010 twitter coverage.

Over at NYC Resistor, it was getting cold, and we needed a doorbell so visitors wouldn’t be stranded outside when the building was locked. A standard wireless model didn’t work reliably (the space is on the fifth floor, just out of range), so various members generally resorted to writing their phone numbers on a sign on the front door when they were expecting guests.

Since almost everyone has a mobile phone already, and SMS-based solution seemed appropriate. In order to implement this we need two things:

1. An SMS shortcode
2. A system to notify when the shortcode is triggered

It’s irritating and expensive to acquire your own shortcode, but there are several services that will allow you to use one in exchange for a small fee or advertisements in your messages. TextMarks is my favorite (I used TextMarks for my WhereAmI project). While TextMarks markets their service as a system for mobile mailing lists, they allow you to reserve a keyword and define a behavior (that can include pulling data from a URL!) to occur when that keyword is triggered.

Configuring TextMarks

Sign up for TextMarks and choose a keyword. Configure the keyword to respond with the “First 120 characters on web page”, and point it at the future home of your script (you can always come back and modify this later).

Note the \0 as the value of the msg parameter — this instructs TextMarks to send along any additional message contents as the value of that parameter. That means if someone were to text 41411 “doorbell hi this is hilary”, TextMarks would call the script with the param msg=hi this is hilary. This can be quite useful.

The Script

This script is written in Python, but you can use any scripting language you like. This particular script just sends an e-mail to an account when the ‘doorbell’ is rung, but you could have it do pretty much anything up to and including ringing a real bell (which may be coming soon!).

#!/usr/bin/env python
# encoding: utf-8
"""
doorbell.py
Created by Hilary Mason, feel free to use this code in your own projects.
"""
 
import sys, os
import smtplib
import cgi
import cgitb; cgitb.enable()
 
class Doorbell(object):
	GMAIL_USERNAME = 'YOURGMAILACCOUNT@gmail.com'
	GMAIL_PASSWORD = 'YOURPASSWORD'
 
	def __init__(self, msg):
		message = """\
From: YOURGMAILACCOUNT@gmail.com
To: YOURGMAILACCOUNT@gmail.com
Subject: KNOCK KNOCK, someone is at the door!
 
%s
		""" % msg
 
		server = smtplib.SMTP('smtp.gmail.com:587')
		server.ehlo()
		server.starttls()
		server.ehlo()
		server.login(self.GMAIL_USERNAME, self.GMAIL_PASSWORD)
		server.sendmail('YOURGMAILACCOUNT@gmail.com', ['YOURGMAILACCOUNT@gmail.com'], message)
		server.quit()
 
		print "You knocked! You can also call us at 347-586-9270. <3, NYC Resistor"
 
 
if __name__ == '__main__':
	print "Content-Type: text/plain\n\n"
 
	form = cgi.FieldStorage()
	if 'msg' in form:
		w = Doorbell(form['msg'].value)
	else:
		w = Doorbell('There is an anonymous monkey at the door.')

And that’s it! Provided you have your keyword configured to point at your script, and the script living at an accessible address, you’ll get an e-mail whenever your SMS doorbell is rung and the person who sent the message will get back a cute response confirming their action.

Finally…

This setup can be easily extended such that a message containing ‘doorbell hilary’ could e-mail only me, or be forwarded to my phone.

I’m curious to see if having a remotely accessible ‘doorbell’ will encourage pranksters — we might need to add a password.

Last weekend Yahoo came to New York for an Open Hack Day, and it was great!

I was invited to speak on a panel on semantic metadata, moderated by Paul Ford (harpers.org) along with Marco Neumann (KONA) and Paul Tarjan (Yahoo/Search Monkey). The panel was a lively discussion, and we got some great questions from the audience.

After the panel, I stayed around to participate in the hack competition. Yahoo! provided a fantastic space, with free-flowing coffee, snacks, comfy chairs and plenty of Yahoo folks and other hackers around to give advice and play foosball with. I teamed up with Diana Eng, Alicia Gibb, and Bill Ward to create the Del.icio.us Cake!

The cake is attached to a laptop via USB. A program running on the laptop accepts a delicious tag and retrieves a list of recent popular sites for that tag from the delicious API. Finally, it iterates through each URL, downloads the page, and computes the sentiment of that page relative to the tag — basically, is the content of the page positive, neutral or negative?

The signal is output to an ardiuno (hidden in the middle of the cake) which turns on the appropriate set of LEDs. There are four sets of LEDs on the cake, one in each quadrant of the delicious logo, one each for positive sentiment, neutral or inconclusive sentiment, and negative sentiment, and, of course, one to let us know that the cake is turned on.

I wrote the sentiment classifiers between around 3am and 6am Saturday morning, so they really were a hack! I trained them on movie reviews data, working with the assumption that 5-star reviews contain positive terms and 1-star reviews contain negative terms. I wouldn’t recommend this approach for a serious attempt at sentiment analysis, but it worked well enough.

We won the food/hardware hack prize, shared with the awesome MakerBot team!

We had a great time creating and presenting the hack. Thanks, Yahoo, and most of all, thanks to Alicia, Bill, and Diana for a really fantastic, silly weekend.

Further coverage:

• Yahoo’s summary of the Open Hack NYC event
• Diana’s writeup for Eyebeam
• CNN.com: Hackers Take Over Times Square

I’ve found myself in need of a name distribution for a few projects recently, so I thought I would post it here so I won’t have to go looking for it again.

The data is available from the US Census Bureau (from 1990 census) here, and I have it here in a friendly MySQL *.sql format (it will create the tables and insert the data). There are three tables: male first names, female first names, and surnames.

I’ve noted several issues in the data that are likely the result of typos, so make sure to do your own validation if your application requires it.

The format is simple:

1. the name
2. frequency (percentage of people in the sampled population with that name)
3. cumulative frequency (as you read down the list, the percentage of total population covered)
4. rank

If you want to use this to generate a random name, you can do so very easily with a query like this:

SELECT name FROM ref_census_surnames n ORDER BY (RAND() * (n.freq + .01)) LIMIT 0,1;

Download it here: census_names.tar.gz

FoxNY did a piece featuring me and Diana as hackers who use our technical powers for good, not evil.

http://www.myfoxny.com/video/videoplayer.swf

There are way too few female technologists on television, and I’m happy to do what I can to show that women kick ass with code! Look for my mischievous I’m-writing-infinite-nested-loops grin in the clip where I’m programming.

If this looks like fun to you, come join us at NYC Resistor (where the segment was filmed!) for Thursday night craft nights or for one of many awesome classes.