Imagining an Open Source Eye in the Sky

In the past, intelligence agencies had the tough job of collecting and aggregating data fast enough and with sufficient relevance that they could have an impact on the decisions regarding the targets of their collection. With the advent of computers, all of that correlation and cross-referencing got a lot easier. With the internet, data sources became enriched, and with modern software and hardware systems, the interfaces to that data have become more natural and powerful.

Businessweek has written about Palantir, a system that aggregates data for anti-terror surveillance.

http://www.businessweek.com/printer/magazine/palantir-the-vanguard-of-cyberterror-security-11222011.html

The history of computers has shown that computing capacity always grows to the point that what used to be affordable to only major corporations becomes affordable to individuals. Today we carry around smart phones with the processing power that entire mainframes possessed in the 1970s. Desktop computers in 2011 have more power than supercomputers of the 1980s.

In addition, the open source world has proven that it can generate powerful software that supplants, replaces, or exceeds commercial applications. Where there is Matlab, there is also Octave. For Excel, there is OpenOffice. For Oracle, there is MySQL.

The next logical step is for open source data aggregation software with capabilities like those inside the three-letter agencies in the U.S., but used and operated by individuals or small organizations. A crowd-sourced intelligence net could field an enormous number of sensors. Most of the pieces of the software have other uses, so it is unlikely that development would be halted from lack of interest. And there are certainly enough people who want surveillance abilities of their own that the audience for this kind of software system exists.

When? Perhaps by 2015. Certain pieces will emerge in 2012. The capabilities for aggregation will grow a bit at a time, with data sockets for spill sources like Facebook, AOL, and Google+.

At the natural maturation point, the open-source software for surveillance data aggregation will match in quality and capability what is supplied to the Government by contractors.