Bomber Hack Project: CommRater

The CommRater site was originally Aviana’s idea which she came up with one of the many times we were procrastinating in the updown this past year. We talked about the project sometime before (or at the beginning of second semester) but development didn’t start until this summer. I really wanted to start during the year but got caught up with work and became too lazy to start a side project. In the end I think this was the best decision since it resulted in a much higher quality website being built by Max and I.

At my summer internship the project I’m working on involves using Django so we thought it would be a good idea to make this a Django project. Neither of us had ever used Django before but we decided that since I had to learn the framework for work anyway. Django has many built-in features that something more lightweight like Flask doesn’t offer out of the box which also made it a good choice. We began the project towards the end of June (or sometime thereabouts) and have been working on the site almost every night ever since. We committed most of our nights after work and a fair amount of our weekend time the past few weeks. The site is now live but is still very much in development right now.

The basis of the site is fairly simple. People have to write comm.prods (funny snippets at the end of emails to the floor) and there was no quantitative way to rate the quality of these comm.prods The site initially started as a simple voting system of up and down votes for each prod allowing the floor to collectively decide which prods were actually good or bad.

As time went on, however, Max and I decided that the site should be much more than a rating system and should really be a home online for Bombers. I’ll briefly outline the systems that we have in place currently as well as our vision for what the site is to become in the future. If you are (or were) a Bomber and you want to register email us at: comm.prod@mit.edu and we’ll hook it up.

Scraping:

The basis of the site is being able to take prods from the floor’s email and post them to our site. This functionality took a lot of trial and error due to dealing with the nightmare of email parsing. I must of dumped the database 100 times before we got the regexes and email thread splitting functionality working at the level that we wanted it.

The basic flow is as follows:

First we added a gmail to the floor mailing list so that we could have a reliable place to get the emails to the floor from. We then setup a MIT XVM machine running a cronjob to ping Gmail every minute and check for new mail. If mail is received it is parsed and posted to the website so we have (almost) live data.

The overall process is fairly simple however the details were hard to get right. The parser accounts for the hundred different ways that email clients decided to make threads (we need to chop off the old content to avoid duplicate prods) and all of the ways that people write ‘ a btb “content here” comm.prod’. It was a fun process and definitely helped me get more savvy with regexs and parsing. Another complication was that people send emails to the floor from multiple emails (MIT mail, Gmail, etc) so we setup a ‘half’ user profile for this case and give users the options to claim these account which merges the two accounts into one.

UserProfiles:

Each user has a profile page in which we display some basic stats of how your rank against the floor and against your class. We also offer the ability to filter the user’s prods with filters such as Recent, Best, or Worst. Each user also has a score which is, based on the number of upvotes for commprods and corrections (discussed below) that they submit.

Correction System:

The parsing problem is one that will never be perfectly solved. Users can write their email in such a way that it is incorrectly parsed leaving wrong content or missing content on the site. To help fix this I built a correction system that allows users to try and fix the parsing mishaps. The basic idea is that the problem can be fixed by crowd sourcing. Users can submit a correction which can then be upvoted and submitted (a correction needs 5 upvotes) or downvoted and deleted (5 downvotes). This is probably the least used feature on the site right now except by admin use but with V2.0 it is more accesible for users.

Display:

The display of commprods is largely influenced by Max’s experience working at Twitter (he copied some of the CSS directly 🙂 ) giving each commprod a “tweet” look and feel. Each prod has a voting option, current score, author and content. In addition you can hover for a snippet of the original email content or click to get the permalink page to see the full content or submit a correction. Commprods containing media content (a url, image, or youtube video) are displayed inline so that the prod is fully shown during the render. The latest release will also allow users the favorite commprods so you can go back to ones that you like.

Trending Score and Data:

Another way to sort prods is by trending score. Each vote adds a fixed amount to the trending score and each minute (with the ping to Gmail) we subtract a fixed amount off of this. This allows us to sort on trending score and display recently voted on prods to users.

In addition to trending scores, each day we record a trenddata point for each user. We basically take a snapshot of their current total score and store them so that we (in a later version) we can show floor trend graphs of user performance. Currently we use the trend data to compute your 30 day trend score.

Future:

The latest release features a whole new set of commprods (thanks to Chris Post) so the site has data from 2005 to present. Max is working hard on a recommendation algorithm so users can be targeted with relevant prods (which is important as the dataset grows and spans many years). Some other new features that I have been working on include favoriting of prods and sorting by prods that contain media content as well as some internal admin tools and optimizations.

Ultimately our goal for the site is to be more than just a way to display and rate prods. We have a few people that want to help develop and we hope to continue working on the project during the school year. Some ideas that have bounced around for addition to the site are as follows:

Text alert system to notify the floor of happenings

Airport/Taxi management system

Floor *shopping* management system

Photo display for prospective MIT students to see what it’s like to be a Bomber

Donations page for alum to give back some love (by that I mean money 🙂 )

Currently the site is up at: burtonthird.com

Sorry, but it’s not public. If you want access (and are a bomber) let us know at comm.prod@mit.edu

Our code is up at, check it out: https://github.com/cantbloom/comm.prod

Bomber Hack Project: Hungry4

Hungry4 was the first project that Max and I worked on this summer and was built during the Hack To the Future hackathon. This was my first hackathon as well as my first node.js project and was really fun to work on and spend a day building. Max and I had talked about the idea a few days before the hackathon and discussed some basic details of the implementation, however, we started and finished everything (except a few tweaks) over the 24 hour period during the hackathon.

The basic idea is that it is hard to find what you are hungry for. Hungry4 directs you to tasty food based on your location (and willingness to travel). Max wrote most of the frontend and recommendation algorithm while I worked on the backend and pulling the data.

The entire app is a single page which start here getting your basic location information:

Based on your search query we hijack the Foodspotting API and query their site for results for your location. We contacted Foodspotting multiple times asking for access to the API but they only replied with “It’s private, sorry” so we just emulated their own requests to get back the data. The call to Foodspotting provides the high quality photos for you to browse and metadata about the dish (name, restaurant, etc) and help us direct you to whatever you’re Hungry4.

The results let you upvote or downvote an item until you find something that you’re satisfied with. We rated each dish in 8 different dimensions [i.e. dinner, breakfast, lunch, hipster, spicy, etc] by dividing the number of Google search results for the dish’s name by the number of results for the name and each different dimension. Then, giving each user a starting standard user vector, your vote would change your vector directing you towards (or away) certain types of dishes. The entire site is stateless, we hold no user information and didn’t user a database of any kind. When the server is first started we cached results for San Francisco since that was our most common query and we would preload the next group of dishes to display to reduce loading times.

If you didn’t find anything you can search again and restart the process. Clicking the “Show more info” button would give you a Google Maps view of the restaurant as well as Yelp information and the option to Tweet or post to your Tumblr about your find.

Overall this was a really fun project to work on and gave me a better taste for web development (this was my second web project). It was also the beginning of the many hack nights that Max and I have done over the summer building the CommRater site.

The website is up at: http://hungry4.herokuapp.com/

And our code is public at: https://github.com/cantbloom/Hungry4

Bomber Hack Project: DTYD ’11

A large part of Bomber culture is the hacking culture. Whether you are exploring MIT buildings at night or building stuff in your dorm room it is certainly a Bomber tradition to hack on whatever you can in your spare time build something. That being said, for the next few posts I want to discuss a few cool hacks that I have been involved with during my time on the floor.

The first hack that I was a part of was the freshman hack for DTYD ’11. Each year before DTYD the freshman class has to promote the party in a unique way with their own hack. My class decided to build a giant replica bomber shirt for the ‘The Alchemist’ a.k.a. the math man shown below:

David Wise and I decided to take charge and began a search across Cambridge for supplies. This involved buying out a local fabric store from all of the orange fabric that they had and several trips to Home Depot. David and I made some rough measurements (mostly by seeing how much taller or longer than our arms the dimensions had to be) and began the long process of learning how to sew and sewing the shirt together.

Most of the class helped with the building of the shirt — mostly building stencils and spray painting the designs on the shirt. After about a week of preparations we were ready to go and deploy. We had no idea if the shirt would actually fit we only went off our rough estimates.

No deploy is good without some hiccups. While we were deploying the MIT police came by and told us we had to stop. There were about 4 people on the deploy team so I had them keep working and tried to talk to the cops so that they could keep itl going up. Overall the cops were very nice and I showed them an email saying that we were *allowed* to put up the hack. Without further incident this was the final product with David and I:

Here is the back of the shirt from The Tech the next day:

The hack was vandalized a day or two later by unknown persons and the shirt was never recovered. The stencils lived on and helped make a mural of the plane in the 41 suite.