Beautiful Data: The Stories Behind Elegant Data Solutions [Kindle Edition]

"Beautiful Data" is a collection of essays on data; how people have transformed it, worked within its confines, and offers a glimpse of where we might go. Many of the essays are wonderful snippets into how some people perceive data while others fall flat. Overall its a mostly enjoyable read that helps open up your mind to new potentials.

First a disclaimer; I am not a data person. However I've been involved, fairly heavily, in the data field. In the parlance of the world, I'm a back end person. However I'm always trying to think about the front end; how will things be used and what information can we gleen from the system (or systems). With that in mind, this is a book that speaks to me - its all about the front end.

Some of the best essays in the book would be:

The first essay by Nathan Yau he talks very much about user created data and personal databases (knowledge bases). What's exciting here is how he takes data already out there, data you have provided, and creates something useful and yes, beautiful, out of it.

The Second essay by Follett and Holm really gets down to how if you want the data, you need to present it in a way that brings people into the process. As someone who has a slight crush on the statistics and practices in polling (and designing poll questions) this essay really was a fascinating read.

The third essay by Hughes detailed how he handled images on the Mars mission. There wasn't anything here that wasn't done in embedded systems 15 years ago; still it was a great walk down memory lane since I used to program embedded imaging systems.

Chapter 4 really hit home PNUTShell is cloud storage and data processing in real time. This really is the stuff of the future.

Chapter 5 by Jeff Hammerbacher really didn't offer too many insights but his writing style is fluid and fun plus he offered a glimpse into how Facebook grew.

We then have the slow section of the book - Chapter 8 on distributed social data had promise but it read more like a company white page than an interesting article. Same with Chapter 12 [...].

Thankfully chapter 10 on Radiohead's "House of Cards" video was there - and here we are presented with true beauty in data - beautiful enough to create a music video out of!

I'm still on the fence with Chapter 13 - What Data Doesn't Do. It was an interesting chapter but it felt both too long and too short at the same time. I almost felt that in the author, Coco Krumme, were to write a book on this topic, I'd want to read it. However her essay was not the right vehicle.

Finally, the last chapter - "Connecting Data" was a truly inspiring piece; one that offers up paths for the future. I am sure a few start ups will form over the questions posed in by Segaran (or maybe the questions to the questions).

Overall there were enough strengths to overcome the weak chapters. My main complaints are trivial; poor binding of the book, too many PhD candidate papers and not enough from out in the trenches. I'd love to see something from Stonebreaker here; its hard to talk about beautiful data and not have him in it. Or forget [...]and talk about many eyes. Or map reduce. Still, "Beautiful Data" succeeds. It opened up my mind to different possibilities for data representation and usage.

... Contents are less impressive. O'Reilly bring together a heterogeneous group of authors and let them fend for themselves, with no editorial effort to unite their stories. Some authors hold their own, presenting interesting analyses and visualizations, or just interesting tales, others are less successful. (The spectrum of statistical expertise, for example, is bounded by Andrew Gelman and a graduate student believing that normality is a requirement of the central-limit theorem). 'Interesting' is a good thing, but for $40 I would like 'useful'. An appealing leisure read, but not much more, I am afraid.

This book tells you what's possible now and what's on the horizon when it comes to data representation, collection, management, processing, analysis, sharing, and display. Very little code is provided because each chapter is mostly a conceptual discussion of approaches to tackling various kinds of challenges involving data, the lifeblood of any application. My favorite chapters are: 4, 5, 7 and 20. Below are my short notes for each chapter to give you some idea of the book's contents.

Ch. 1 Seeing Your Life in Data by Nathan Yau
Hoping to better understand their impact on and exposure to the environment, participants in one of Yau's projects download software onto their phones that then upload GPS data to servers as they go about their daily activities. One of Yau's early challenges was to summarize the data and make it meaningful to the participants: for example, what does it mean to emit 1,000 kilograms of carbon in a week? What he found helpful and not so helpful in data visualization are instructive.

Ch. 2 The Beautiful People: Keeping Users in Mind When Designing Data Collection Methods by Jonathan Follett and Matthew Holm
When there is no explicit profit to be made, how do you convince a person to take the time to answer your survey questions?

Ch. 3 Embedded Image Data Processing on Mars by J.M. Hughes
Like everything else onboard a spacecraft, the computing system is custom built with minimalism and other stringent specifications (e.g., withstand radiation) in mind. How does one harness limited resources to get the job done?

Ch. 4 Cloud Storage Design in a PNUTShell by Brian Cooper, Raghu Ramakrishnan, and Utkarsh Srivastava
Yahoo! engineers have a very challenging job. Web pages containing potentially complex social data must load and update quickly regardless of where the data may be mastered in servers distributed across the world. Learn why they jettisoned some conventional database concepts in favor of: flexible schemas, timeline consistency-driven data updates, etc.

Ch. 5 Information Platforms and the Rise of the Data Scientist by Jeff Hammerbacher
The author mentions that according to IDC, the digital universe will expand to 1,800 exabytes by 2011 (1 exabyte = 1 billion gigabytes) and the vast majority of that data will not be managed by relational databases. The Facebook Information Platform described in this chapter can manage structured and unstructured data in an integrated manner, and can extract useful information from terabytes of data in seconds. Similar platforms built at Fox Interactive Media and Microsoft are also described briefly.

Ch. 6 The Geographic Beauty of a Photographic Archive by Jason Dykes and Jo Wood
The Geograph British Isles Project aims to collect geographically representative photographs and information for every square kilometer of great Britain and Ireland. Learn new data visualization techniques!

Ch. 7 Data Finds Data by Jeff Jonas and Lisa Sokol
Technologies similar to those already used in, say, fraud surveillance can be adapted for other more mundane applications.

Ch. 8 Portable Data in Real Time by Jud Valeski
How can companies facilitate the sharing of and access to social data without having to invest on an inordinate amount of infrastructure?

Ch. 9 Surfacing the Deep Web by Alon Halevy and Jayant Madhaven
Web contents that lie hidden behind HTML Forms are part of the Deep Web that search engines have not indexed very well but that may partially change soon.

Ch. 10 Building Radiohead's House of Cards by Aaron Koblin with Valdean Klump
The author helped produce a video for the music group entirely from visualization of data, and without the use of cameras or lights. Google Code urls given. You gotta see the interesting video!!

Ch. 11 Visualizing Urban Data by Michal Migurski
Learn how to visualize trends in urban crime, using maps and data mashups

Ch. 12 The Design of Sense.us by Jeffrey Heer
The combination of interactive visualization and social interpretation can help an audience more richly explore a data set.

Ch. 13 What Data Doesnt't Do by Coco Krumme
Data doesn't stand alone. In real-world decision-making, information is rarely packaged neatly and data isn't free from interpretive biases.

Ch. 14 Natural Language Corpus Data by Peter Norvig
Natural language tasks like word segmentation or spelling correction can be handled using probabilistic models built from processed large data sets.

Ch. 15 Life in Data: The Story of DNA by Matt Wood and Ben Blackburne
The human genome has been well annotated and 40 other species have been sequenced. With each new discovery, however, more questions are raised, and more research data is generated. The need for efficient sequence search, alignment, and assembly tools, as well as safe housing for the millions of genomes, will continue to grow. Learn how scientists are rising to the challenge.

Ch. 16 Beautifying Data in the Real World by Jean-Claude Bradley, et al.
How online publishing of scientific data can be improved upon

Ch. 17 Superficial Data Analysis: Exploring Millions of Social Stereotypes by Brendan O'Connor and Lukas Biewald
Ch. 18 Bay Area Blues: The Effect of the Housing Crisis by Hadley Wickham, Deborah F. Swayne, and David Poole
Ch. 19 Beautiful Political Data by Andrew Gelman, Jonathan P. Kastellec, and Yair Ghitza
These chapters show you data analyses in action: how to prep data, smooth out the effects of noisy or outlier data, etc.

Ch. 20 Connecting Data by Toby Segaran
We need to break down information silos but how? The use of Semantic Web and/or Collective Reconciliation techniques are discussed.