Jeffrey Veen

Video: Designing for Big Data

This is a 20-minute talk I gave at the Web2.0 Expo in San Francisco a couple weeks ago. In it, I describe two trends: how we're shifting as a culture from consumers to participants, and how technology has enabled massive amounts of data to be recorded, stored, and analyzed. Putting those things together has resulted in some fascinating innovations that echo data visualization work that's been happening for centuries.

I've given this talk a few times now, but this particular delivery really went well. Only having 20 minutes forced me to really stay focus, and the large audience was very engaged. I'll be giving an extended version of this talk in June at the UX London conference, with a deeper look at how we integrated design and research while I was at Google.

 21 Apr 2009 | Comments (9)

Announcing Wikirank: Tracking what's popular on Wikipedia

A few months ago, the four of us in Small Batch Inc were kicking around ideas for what we should build next. We had just launched the Twitter Election site -- a fun real-time data visualization project -- and wanted to keep building things that help people make sense of the enormous amount of information that bombards us each day. Late one night soon after, I followed a link to a repository of Wikipedia server logs. There were gigabytes of data sitting there just begging to be visualized. We got to work.

The result is Wikirank, a tool for exploring what's popular on Wikipedia, discovering comparisons between topics, and sharing them with the world. We launched it yesterday, happily coinciding with the 14th anniversary with Ward Cunningham's invention of the wiki.

There are a bunch of reasons why I think Wikirank is cool, but my favorite is how it helps people find stories in the data. One of the great things about the web is how measuring tiny behaviors reveals patterns that tell stories. The data we get from Wikipedia is no different; as we started playing around with the numbers, we saw loads of interesting shapes emerge in the charts.

For example, big news stories show up as dramatic spikes where there there was no data before. When the astounding story broke that an airliner had made an emergency landing in the Hudson, a page was created on Wikipedia within minutes. Over the next two days, that page was one of the most popular on the site.

The shape takes a different angle as we watch the marketing buzz and fan excitement build towards the release of the Watchmen movie earlier this month.

Comparisons are where Wikirank really shines, however. The weekly viewing habits of television watchers comes into clear view -- as do the day on which the shows air -- when we compare Heroes to Lost. (It's also my guess that the occasionally perplexing plotlines of both of those shows leads a fair number of people to Wikipedia to find out what the heck just happened.)

I'm really pleased with how this project came out. I'll write more about the technology behind the project in a followup post, but the reality is this launch represents a pretty big shift in how we build web apps. With only a couple developers and a rack of rented machines in the cloud, we pulled this all together in just a few weeks. That simply wasn't possible the last time we did this.

The team behind all of this includes my long-time friends and business partners Bryan Mason, Ryan Carver, and Greg Veen. We were also extremely fortunate to be able to work with Dan Cederholm of Simplebits, who helped us with visual design and identity. Dan rocks.

We'd love to know what you think. You can join the discussion at Get Satisfaction.

 26 Mar 2009 | Comments (15)

Conference hack: Embracing the backchannel at Start

I had a very interesting experience a few months ago while participating in a panel discussion. Once again, I realized that the content on stage is merely the spark of a broader conversation, and that the backchannel is rapidly becoming the whole point. So we've decided to try an experiment at the Start Conference in a couple weeks to see how we might hack traditional presentations.

But let me back up a bit first. I was on a panel at this year's South by Southwest talking about the role of analytics in design. With me were two veterans of the advertising industry who's work included some of the biggest ad campaigns of the past few years - some really amazing stuff. I started the conversation by saying how the remarkable amount of audience data available to us gives designers tremendous power to affect user experiences. My collegues suggested my approach sucked the creativity out of design. I countered that they were mistaking preferential research from behavioral. The argument heated up.

While this was happening, my phone was buzzing non-stop. I slipped it out of my pocket to discretely turn it off, but noticed a stream of Twitters going by - many from audience members in the room. So I set the phone down on the table in front of me and kept an eye on it. I'm so glad I did.

As the conversation on stage continued, the stream of questions and comments from the audience intensified. I changed my tactics based on what I saw. I asked questions the audience was asking, and I immediately felt the tenor of the room shift towards my favor. It felt a bit like cheating on an exam.

I guess it really wasn't cheating, but it does illustrate one of the frustrations I've had at conferences lately. Most of the events I attend have a rich conversation happening in the room, yet the only people not able to participate are those on stage. A couple times, I've seen organizers project a live IRC channel, but that usually bring out the worst in people ("First!!!111") - and is terribly distracting. So I've been wondering for a while if there was something smart we could do at our conference.

Apparently, Bryan had the same idea. As we were planning Start, he said, "We should have someone onstage the whole time to represent the audience. Like an ombudsman does for a newspaper."

So we decided to put a desk on stage and have our friend George Oates fill that roll. She'll be on Twitter, IM, and email listening to what people are talking about. (We'll also have volunteers collecting index cards for those not wanting to be online during the sessions.) And she'll synthesize questions, interrupt us if we get boring, and call bullshit if something sounds like it. George has been the designer at Flickr since it started; her personality has always shown through there, and will be a great fit for what we're trying to do.

What do you think? I've never really seen a conference do something like this before. Have you?

 25 Jul 2008 | Comments (18)

Robot Cross Stitch

This was a project I started a couple years back that I lost between moving, packing and unpacking. I found it a few days ago and finished stitching it over the weekend. It was inspired by my favorite t-shirt back then.

I like cross stitching for it's obvious connection to pixelated icons and images from the web. I made the pattern by grabbing a picture of the shirt, knocking out the background, and posterizing the colors down to just red, yellow, and black. Then I blew up the image 400% and added a grid to show each stitch. Now, you can just upload an image here and have it spit out a pattern. You'll still need to spend time simplifying and sizing the source file, though.

I've always liked images of robots - especially mid-20th Century anthropomorphic representations. I like that style, but I also like the metaphor. Robots represent our approach to technology and, frankly, how wrong we usually are. We had hoped to create tech in our own image, like Rosie, the Jetson's maid who zipped around the house cleaning and cooking. We hoped that robots would enable a life of pure leisure - walking, talking butlers that do our bidding.

Our present reality is far from those idealistic days, but in a way I'm grateful for that. Instead of technology serving us, we've created powerful tools that amplify our abilities. We use computers to make us smarter, the web as an outboard brain, networks to stay connected. Rather than building robots as better humans, we've used technology to become better ourselves.

But that's just me at my most optimistic. The truth is I like to stitch things like this to get me away from the damn computer. I think I'll start another project...

 22 Jul 2008 | Comments (1)

Who is John Snow?

Last month, while attending the fantastic @Media conference in London, we paid a quick touristy visit to a street corner in the heart of Soho. There, we found a small monument - an old hand-drawn water pump on a stone pedestal, its handle notably missing. We climbed up on it, had our photo taken, then ducked across the street for a pint in an old pub named The John Snow.

The pub's name was interesting to me for a couple reasons. I had just finished the Steven Johnson book, "The Ghost Map," a riveting account of the neighborhood's 1854 cholera outbreak. Johnson tells the story of Dr. John Snow, who had lived near where we had our picture taken, and how he followed a hunch that the disease could be spread by water, rather than being airborne. After days of investigation, Snow proved to London's health commission that this Broad Street well was the source of the outbreak, convincing them to remove the handle. It was an early instance of science and evidence winning against superstition and prejudice.

But even more interesting to me was one of the tools he used in his work - a data visualization showing cholera deaths marked on a map correlated to the location of the pump. In the speech I gave that week, I pointed out how his graphic techniques are still a model for good design over 150 years later. Contrary to the popular myth, Snow didn't create that famous map; he was working from an existing visualization first drawn by sanitation engineer Edmund Cooper months earlier. Snow did, however, redesign the work by carefully eliminating elements and emphasizing the casualty data until the narrative of what had happened was perfectly clear. You can't look at that map without thinking, "Yikes! Stay away from that pump!" Snow's early geo-mashup can still inspire.

So back to the eponymous pub. As we stood near the crowded bar, I decided to do a little experiment. Even though there was a plaque on the wall detailing the establishment's history and namesake, I started asking the folks around me, "Hey, why is this place called 'The John Snow'?" hoping to find fans of the historic story - or maybe even someone interested in the visualization. Instead, here's what I got:

The tall Irish guy with a ponytail: "No bloody idea, brother."

The group of women laughing in the corner: "Oh, he's some wanker on the telly who bought this place years ago." (They were thinking of this Jon Snow, who is actually quite respected.)

The bartender, who came close, but really should know better: "Before this was a pub, there was a hospital here and he was the doctor in charge of it."

How quickly history fades - especially after a few drinks.

  7 Jul 2008 | Comments (6)