Open source intelligence data technical issues and verified open source intelligence

Technical issues with open source intelligence data

When it comes to open source intelligence data, there are at least two types of technical issues involved. One is to identify relevant information, and then retain the original source information in the process of collecting data, and remove other information generated based on the source information; the other is to collect data to facilitate analytical investigations in order to produce credible results .

Regarding the first type of challenge: Although collecting and storing data was expensive in the past, with the advancement of data storage and databases, this problem has become less prominent. As recently confirmed by Edward Snowden, the U.S. government has taken great strides in this regard.

In comparison, the more challenging and valuable problem is how to extract important information from unstructured text information. In other words, how to remove noisy data and secondary data, and only keep the most important information (such as location, threat classification, date, people, etc.). This also means that unstructured textual information needs to be transformed into a relevant data format that can be organized and queried from multiple dimensions.

An obvious advantage of such data is its reusability. Traditional qualitative analysis can only be used to answer a single question at a time; big data can be transformed multiple times and used to answer many different types of questions.

Verified open source intelligence

Verified open source intelligence is open source information with a high degree of certainty. Typically, such intelligence information is derived from the analysis of all-source intelligence professionals combined with classified sources. In addition, it can come from certain publicly available sources, such as media broadcasts showing live video of a plane arriving at an airport.

Example: A CNN reporter takes pictures of a bridge and writes a report. Intelligence agents knew the bridge was important for the rebel group to move supplies, but CNN reporters did not. Classified information suggests that such a bridge exists. In effect, this amounts to confirming the CNN report as verified open-source intelligence. Obviously, once the bridge proved destroyed, it would be very difficult for the rebels to move supplies. In the absence of any other information that the bridge was destroyed, "the bridge exists" can still be considered verified open source intelligence. This is especially true if there are other sources of information (such as other information reports) attesting to the existence of the bridge.

It can be said that before raw data can be called "intelligence", it must be analyzed. If possible, preferably verified. The CIA alone has about 2,500 people dedicated to this work. Thousands of people are dedicated to the work, including across all 16 intelligence agencies.