Findability

Peter Morville's book Ambient Findability crystalized much of my thinking about organizing data. Data is worthless (but expensive!) if you don't know how to find what you want. Data needs to be structured to answer questions.

Data warehouse projects that are not designed for findability can easily turn into data land-fills. I observed one project where a team spent a year scraping websites and documents collecting all the field trial data they could find. The no-SQL programmer on the project convinced management that no-SQL focused on collecting data and the retrieval issues could be added later. Concerns that one project recorded fertilizer as pounds per acre, and another categorized plots as 'high, medium, low' were dismissed. Inconsistency (or absence) of location information was equally brushed over. That variety IDs could not be tied back to gene sequences was not a concern. The goal was to collect as much data as quickly as possible. It wasn't till over a year later that they tried to run a report, and discovered that while they could retrieve documents, they could not correlate one document with another. They then discovered that the cost of collecting data is much lower than the cost of cleaning up data, or identifying which data should be deleted.

Findability starts by asking how the data will be used, what questions will users be asking. Some questions are easy and obvious, but often questions will be asked years later that were never considered by the original project. I have a valuable sets of bathymetry (depth) data for Lake Monroe that is a set of transects by the Army Corp of Engineers to determine sampling locations for a mercury pollution study. These have become the standard locations for collecting further bathymetry data used to evaluate sedimentation. I am sure that sedimentation was not on the minds of the people who originally collected the data, but good data increases in value as it gets re-used in new ways.