U.S. Federal Government Data

I keep on being pleasantly surprised at how much data U.S. federal and other government agencies make available to whoever wants it.

Over a decade ago, when I was doing Artificial Intelligence research, DARPA, the Defense Advanced Research Projects Agency kindly sent me some CD-ROMs with data that they hoped AI researchers would find useful. No strings attached, and shipped to a Canadian address. More recently, NIST has done some of the same things for people involved in biometrics. NIST also has a large number of other data sets that are made easily available to universities, but which private companies have more problems getting at. Still, very useful data at a reasonable price.

The state of Florida provides so much data on businesses and business transactions online that it's almost embarassing. Whenever I deal with a Florida firm I check them out. I stop short of intrusive things like the mortgage on the house of the individuals I'm dealing with, but all information the state has on the firm and its principals is there, online and free.

I am now working with housing and household transportation data. I had found that agencies like HUD are generous with giving away data online, but now that I am working with household travel data, I find that not only are large amounts of extremely valuable travel survey data are available online from the U.S. Department of Transportation, but they even provide fairly sturdy data processing horsepower to let you manipulate their data. Gratis, in aid of research.

A friend of mine was interested in unreleased NASA data. All of it released to him, with gratitude that it is being used. He is now providing them with some software he developed to process this data.

I am amazed and grateful at the lack of bureaucratic obstacles to stop people from making use of the major assets that the government collects. I know of very few Canadian government data sources that are so generous. Sometimes small samples are available for free but for any significant use of data you have to pay or hunt down and sweet-talk the appropriate public servant. We would have higher quality of research and development in Canada if governments were to routinely release all of their data to the public, where it is likely to be of more use than locked up in government databases or sold through some agent. Many public servants in Canada collect data or develop intellectual property that is never used by anyone. A "cost recovery" price is set and seldom collected. In my opinion, virtually all of it should be released as open source or even put in the public domain.

References
http://www.transtats.bts.gov/
http://www.huduser.org/datasets/pdrdatas.html

[] permanent link Comments: 0