A personal history of the effort to find the survivors of Hurricane Katrina

This is not a dramatic story from Hurricane Katrina. There are no jumps from helicopters to save victims being washed away in raging waters. There are no stories of deciding who lives and who dies in triage centers. This is the personal story of one person's effort to share heart wrenching information and build a community on the Internet to help the victims of Hurricane Katrina.

The are a thousand stories of courage, survival, and great loss in Hurricane Katrina. Why tell this story? This story aims to explain how an effort to share information openly attracted people around the world to help in a chaotic and a surprisingly effective way. Right now, the people finder project, have shared almost 400, 000 survivor records and written over 1200 emails on the KatrinaDev list alone. I have recieved a lot of feedback from people who have looked at this effort and seen both chaos and inspiration almost simultaneously. The purpose of this personal story is so that in the future more people will understand what we did, and encourage people to contribute to efforts to help each other in this new and interesting way. It's too early to tell an accurate story of what the people finder project did, but for now I can tell my story leading up to it.

The passion to help the victims

The effort to help the victims of hurricane Katrina was successful because of the passion of the volunteers who contributed to it. The people who contributed to the hurricane katrina people finder effort were passionate about helping the victims. That passionate focus on the victims throughout the project allowed us to stay focused and tolerate the failures and setbacks. The passion of the highly skilled individuals who contributed to organize this community and create technology was key to our success.

While there were many projects to help gather information about the survivors of hurricane katrina there were two key differences about our project. These differences are cultural by products of the open source movement and grass roots community organizing.

The first difference was that were able to innovate because we failed quickly and cheaply. Corporate project management teaches us that we must effectively use our limited resources and manage risk. In this project we encouraged volunteers to contribute resources in any way they wanted and assume that most would fail, but the best ideas would attract more resources. We were right.

The second key difference on our project was that we were open and shared ideas. While we certainly failed to partner with key projects such as the Katrina data project and the Red Cross early enough, we managed to share ideas in an open way and take the time to iterate. A key to this was the decision to create and share the People Finder Interchange Format.

A personal history leading to data sharing

I am going to explain briefly my history with data management and how this led to the creation of an effort to share the list of survivors from Hurricane Katrina. I want to share this history because it's important we understand that these moments of innovation take investment in research, years of practice, and a culture of sharing and openness.

My first computer was an Apple IIe, which my step father used for home programming and teaching us about computers. My step father is a database designer and has coached me on database analysis and design for many years. During university I took several internships, which included data analysis and database application programming. My first job out of university was as a C++ programmer on a large clinical medical record project. One of the worst experiences on this project was watching the team implement an object serialization library to help extract object data into relational data. I started studying this object to relational problem and consulted with my boss about the importance of teaching the customer to understand what data is important for reporting and what data is not. This experience made me realize that if customers do not know the true value of their data, they will wrongly focus on getting all of it instead of getting value from it.

My second job was as the technical lead for an innovation laboratory located at IBM's research facility in San Jose. One of my first projects was working with Hamid Pirahesh, an IBM Fellow and leader in relational and XML data research. The project was to convert multi-dimensional data, called data cubes, into an XML Cube format to allow data to be shared across the Internet on mobile devices. While the project was not a success I learned a lot about translating data from structured data sources to semi-structured sources and vice-versa. But more importantly, Hamid taught me the value of failing quickly and cheaply. I realized that big companies were so well managed that they had forgotten to teach people how to fail successfully. This realization that there was value in failing quickly greatly influenced my thinking about how to approach projects.

At IBM I continued to pursue risky, research projects that focused on learning a lot by failing quickly, and cheaply. I realized that it was better to try a dozen ideas passionately than to pursue one idea cautiously because real innovation was not predictable. The key was to re-combine lots of good ideas in new and novel ways. The added benefit of trying lots of ideas passionately was that it helped you build a network with great people. That network would become critical for future projects.

I was also increasingly becoming interested in open source software and how it supported a meritocracy of good ideas. While failing cheaply was attractive, failing freely seemed like the ultimate model for innovation. Also, since these project were open source there were no limits to the number of great people who could network their ideas together.

Joining CivicSpace Labs

I joined CivicSpace Labs in January of 2005. Upon joining I immediately did a quick business evaluation and realized that while there was value in creating great communities, there was even more value in creating a network of communities. In order to network these communities it was clear we were going to need stronger notions of data management and data integration. One of the most important projects CivicSpace was working on was the integration with the CiviCRM project. I began researching integration with CRM, customer relationship management, systems. Michael Haggerty, from Trellon, and a friend from the IBM Data Warehouse team both recommended that I research data ware housing guru Ralph Kimball's writings.

The first lesson of integrating data from different sources, that I learned, is that you need to win support for a standard data model that disparate data sources can be mapped to. The second lesson was that we needed stay focused on how the data we were integrating would support business decisions. Both these lessons would later prove critical in the success of the people finder project.

Once I completed my first phase of data warehouse research I began thinking about what this meant to the CivicSpace community. CivicSpace has it's roots in the 2004 presidential election campaign. The anecdotes from the election were that while democrats were focused on new web applications, Karl Rove(Bush's brain) was making political strategy decisions 18 hours a day on his data warehouse of political information. These anecdotes have since been debunked. However, it was clear to me that grassroots communities needed the ability to gather their disparate data and give their community members these executive decision making tools. To experienced technologists this kind of pronouncement usually causes various degrees of physical shock. But I didn't care. I had a clear vision that we needed a common data model, and a focus on helping community members get the data they need to make decisions.

The first step in getting a user data model for CivicSpace was to integrate with CiviCRM. This would allow us to identify what data we needed to share and what the programmatic API CivicSpace would need to expose to support those data needs. CRM systems typically manage three types of data: referential data, relationship data, and activity data. CiviCRM focused mainly on referential data, e.g. personal information, and relationship information, e.g. who lives in the same household. We are now working on integrating community activity data into CiviCRM.

I knew the next great application of user information will probably not come from CivicSpace. But the application would probably be cloned into CivicSpace and we would need to be ready integrate with this valuable information. We had to have a data model in order for this to happen.

Networking for innovation

In July 2005, I attended Advocacy Dev II and met Clint Oram, a founder of SugarCRM. I explained my ideas of coming up with a simple common data model that would allow for sharing of user information between applications. Clint validated my ideas and we agreed to work together in the near future to come up with a common data model for user information. Later that month I was invited to host a panel on Open APIs for the N-Ten non-profit technology conference in San Francisco. My first panelist was Steve Wright from the SalesForce foundation. Clint Oram from SugarCRM and Donald Lobo for CiviCRM also agreed to be on the panel. I now had one open source application platform, CivicSpace, and 3 CRM systems all talking about open APIs and how we can integrate and share data. We got good feedback on the panel, and it was clear there was a demand for stronger integration of people data.

At the same time, application integration with CiviCRM was taking off. We were integrating mass mailing, volunteer, and organic group creation with CiviCRM. There was also work being done on petition, and event finding integration by Michael Haggerty, from Trellon.

Preparation before opportunity

For me there were several pieces in place. I believed that passionately implementing new ideas with a diverse group of people was the best way to innovate. I had a desire to integrate distributed user data sources via a common data model. I wanted to integrate this data in a way that would support community members goals. I also had a network of people who wanted to work together and they had the resources and tools to do so.

Hurricane Katrina

I was in Iceland at a bachelor party when Hurricane Katrina happened. When I got back I was sick and trying to push out the latest release of CivicSpace, the one integrated with CiviCRM. On Thursday morning, Molly Neitzel, the executive director of Music for America contacted me and told me that the League of Pissed Off voters of New Orleans had been evacuated to Houston. She wanted to know if we could help them. I quickly got in touch with David Taylor from Radical Designs who was providing technical support for the League. They were trying to get requirements from the league. At 1PM on September 1st, I started recruiting people from the CivicSpace community to help develop a CivicSpace site for New Orleans network. I got a lot of volunteers to step forward, in particular Lynn Siprelle and Herman Webley. David Taylor called me back later that day to let me know what the league needed most was a way to find the people in their neighborhood.

I contacted David Geilhufe, Donald Lobo, and Dave Greenberg to ask if they would modify CiviCRM to help with the New Orleans Network people finding effors. The CiviCRM team took off with a passion. When I got an email the next morning from Dave indicating they were having a lot of fun, even though we were have serious hardware issues I knew we were on to something. Zack Rosen, from CivicSpace started networking like crazy and put me in touch with Ping Yee, who was well known for his work helping to find people after September 11th. David Geilhufe sent me an email around 5AM on Friday saying he had been thinking about phase II. On Friday September 2nd at around 9:20AM I emailed Steve Wright from the SalesForce foundation, and Clint Oram from SugarCRM. Both got back to me immediately. By 12:40 Friday after a phone call with Steve Wright, David Taylor, and David Geilhufe we had an initial set of fields for the People Finder Interchange Format specification. Zack and I agreed we needed a mailing list to extend beyond just the new orleans network project. Ping, Jon Plax from salesforce, and I went back and forth for a while and by 9:40 PM Friday evening Ping released an initial draft of the People Finder Interchange Format. Jon Plax committed to support the spec early on behalf of Saleforce. We now had CivicSpace Labs, CiviCRM(Social Source Foundation), Ping Yee, and Salesforce working together on a common specification to share data across all our sites. We had an agreed upon data model and all had a clear business goal for the data we were sharing, to help the families of the victims find each other.

I was so busy working to help out, it took me until noon Saturday to subscribe to the katrinadev mailing list. The first time I checked my email there were dozens of messages on the list. Things just took off from there. The rest is, as you say, history.

Personal Stories

The is just one of thousands of stories of people who made the people finder project happen. There are still many great stories to be told of what has happened and how it happened. I still have no idea where Jon Lobkowsky and the 250 volunteers on the peoplefinder list came from. I hope there is an aspiring story teller out there who will look through all the emails and the cellphone logs to tell this story more completely. I hope this story makes it easier for people to understand how so many people can come together and work through the chaos, and public failures, try so many things, and in the end succeed in helping the victims of hurricane Katrina.