The dilemma of open source threat intelligence in cyberspace

With the rapid development of network and information technology, the value of open source intelligence in strategic investigation and analysis is becoming more and more significant, and the way intelligence plays a role is also constantly developing and enriching. Open source intelligence uses artificial intelligence to aggregate scattered data traces into high-value knowledge fragments, thereby providing deep insights and insights into the situation reflected by information.

Cyberspace open source intelligence is a subset of open source intelligence that focuses on information reflecting attacker tactics, techniques, procedures, behaviors, events, and all other elements of value to cyberspace defenders. Appropriate, efficient, and timely cybersecurity threat intelligence helps identify what is happening, why it is happening, and how to deal with the risks.

In the era of big data, the acquisition of open source threat intelligence in cyberspace faces the dilemma of "data explosion" but "knowledge scarcity". Threat intelligence sources can be scattered across social networks, blogs, Twitter, news sites, forums, and many other venues, and the number and frequency of updates continues to increase. This unprecedented amount of data has brought unprecedented difficulties to threat intelligence analysts to complete the workflow of "observation-guidance-analysis-output". When the complexity of data volume and data association relationship exceeds their understanding and control, a cognitive crisis will be triggered, which is mainly reflected in the following four aspects.

1. The credibility of threat intelligence is questionable.

Any security researcher, user, hacker, or government employee may post any content on the Internet, regardless of their academic background, judgment, beliefs, or intentions, and the quality of such content cannot be guaranteed. This is especially true when intelligence analysts lack effective ways to distinguish fake data from real information, especially when large amounts of such data are obtained in a short period of time by means of web crawlers or database downloads.

2. The integrity and consistency of intelligence cannot be guaranteed.

Threat intelligence can be generated from a variety of channels or sources, including human experts, devices, or automated response programs, which may not have a clear organizational, objective, or administrative purpose. As a result, the information available to analysts on a topic always comes in a disjointed, fragmented, and contradictory manner, and it is difficult to draw meaningful answers from this mess of data.

3. The randomness and uncertainty of the analysis process.

Intelligence analysis is a process in which analysts analyze and process intelligence information through systematic and meticulous thinking activities, gain insight into the opponent's true intentions, and predict development trends. In addition to a large amount of intelligence information, the analysis process also requires professional analysis skills, professional analysis tools and rigorous reasoning logic, all of which are closely related to the analyst's personal experience. When faced with the same material, different analysts may even draw completely opposite conclusions.

4. The accuracy of prediction is unsatisfactory.

Improper intelligence collection, insufficient data support, errors in analysis and judgment, and rigid thinking may all lead to deviations and errors in the final research and judgment results. Especially in open source threat intelligence research, analysts can always only grasp part of the information. Even rational and rigorous analysis is prone to errors due to cognitive gaps, information asymmetry, and biased opinions.

The above four problems cannot be solved by simply increasing computing power, improving algorithms and expanding storage power. Faster computing efficiency, stronger recognition level, and more ample storage space can alleviate the situation of insufficient resources, but "people in the loop" is still an important prerequisite and key feature of open source intelligence analysis. Strengthening human-computer cooperation based on human inspiration, intuition, sensitivity, and macroscopic grasping capabilities, as well as high-speed computing, storage, and communication capabilities of computers, can truly improve accuracy and efficiency in the field of open source threat intelligence analysis, and observe from cyberspace the clues in the results extract important information about the attacker, attack behavior, and attack intent. Once a smooth knowledge exchange and sharing path can be formed between humans and machines, and an iterative cycle of autonomous intelligence analysis can be established, this hybrid intelligent system will surely gain a huge advantage in fighting cyberspace attackers.