A new form of intelligence work in the era of big data - Internet Open Source Intelligence

1. What is intelligence?

Early intelligence mostly refers to secret intelligence, serving military activities, in order to win wars and maintain national security. With the development of science and technology and the deepening of social practice, the concept of intelligence is constantly evolving and enriching, the connotation is gradually clear, and the extension is gradually expanding. For example, many modern interpretations of intelligence are made from the aspects of philology, archives, library science, etc., which are extended and expanded concepts. In fact, the most intuitive and simple explanation of the word "intelligence" is: intelligence is the key information needed to solve a specific problem.

2. What is open source intelligence?

United Stated Congress, NATO, Open Source Solutions Academy, U.S. Department of Defense, U.S. Director of National Intelligence and other institutions and individuals have clearly described open source intelligence. In a nutshell, open source intelligence refers to intelligence obtained from public sources and obtained legally.

With the development of science and the popularization of Internet technology, the methods of open source intelligence collection have been expanded in electronic media, and Internet Open Source Intelligence has also begun to take shape. Internet Open Source Intelligence (INOSINT) is based on clear and specific needs, based on public data resources, through legal and compliance means, comprehensive various types of massive data, using advanced processing technology, conducting professional intelligence research and judgment, and mining and refining useful information. The intuitive explanation is: Gain intelligence from publicly available data in cyberspace.

Throughout the world, open source intelligence has been widely used in maintaining national security, supporting military operations, ensuring law enforcement operations, serving teaching and scientific research, supporting business competition, supporting disaster relief, and participating in social governance. Developed countries represented by the United States attach great importance to the research and application of open source intelligence. The relevant theories have become more complete, and the actual intelligence work has turned to open source intelligence. Sherman Kent, the "father of intelligence analysis" in the United States, once said that about 80% of the intelligence used by the intelligence community every day comes from open sources, and Allen W. Dulles, the longest-serving director of the CIA, also publicly recognized this view. According to CIA statistics in 2007, open source intelligence accounted for more than 80% of the total intelligence collection.

In the Internet age, open-source intelligence has a large quantity, rapid generation, wide access channels, and strong timeliness, which are completely incomparable to traditional intelligence. This has also greatly improved the value and status of open source intelligence, and at the same time, the research on open source intelligence has also introduced a new direction - Internet Open Source Intelligence.

The US CIA has established the Center For Cyber Intelligence (CCI) and the Center for Cyber Intelligence Research (CCIR). The British Broadcasting Corporation also has a department called BBC Monitoring, which is responsible for monitoring foreign media around the world, and also uses open source intelligence, especially open source network intelligence, as the main monitoring method. All these show that the Internet has become the world's largest source of open source intelligence data, and the acquisition of intelligence has entered the era of open source network intelligence.

3. What are the characteristics of open source cyber intelligence?

First, the data sources of open source cyber intelligence are relatively wide. It not only covers traditional public media, such as public speeches, public broadcasts, etc., but also includes Internet-specific media, such as social media platforms, emails, online media, major websites, surveillance cameras, communities and other domestic and foreign data sources. It covers almost all fields and industries, and can be used as the "starting point" and "entrance" of another intelligence topic at any time to guide intelligence personnel to discover clues and conduct "full information source" collection and analysis. At the same time, the characteristics of comprehensiveness and stability of open source intelligence are convenient for tracking the changes of the target for long-term monitoring. Comparing the intelligence services of many companies and research institutions in recent years, it is found that using open source network intelligence as the main data source can meet most of the needs of users. Such as research on competitors' technical conditions, business dynamic tracking, management, organization, research and development, and application status research of related equipment and technologies, industry operating cost composition and low-cost strategy analysis, new technology research and development and application status and trends, etc. .

Secondly, the input cost is relatively small. Experts believe that compared with other intelligence, the economic cost of open source network intelligence is lower, with less investment and greater returns. Therefore, for those countries with insufficient intelligence work budgets, open source network intelligence can be used to replace traditional secret intelligence to a certain extent. In addition, traditional intelligence work requires professionals to collect intelligence, and the cost is relatively high. Using Wikipedia and other mechanisms such as Web2.0 can mobilize all personnel in the organization and those interested in the topic in the society to collect intelligence together, greatly reducing costs.

Furthermore, the security risk of open source cyber intelligence is low. For enterprises and social organizations, open source network intelligence can avoid the risks of illegal or moral violations that may exist in other intelligence work. For nations, open source cyber intelligence can avoid the diplomatic wrangling that other types of intelligence work often spark. Since the information comes from the open Internet, there are not too many restrictions and obstacles in the process of collection, analysis, and transmission. Open source intelligence does not involve confidential information, and all work is carried out through legal and compliant means. From information acquisition to intelligence use, the security risks at all stages are relatively low.

Finally, the concealment of open source network intelligence is relatively high. Open source cyber intelligence can protect one's own strategic intent. Traditional intelligence work often requires the use of various manual or technical means to spy on the other party's system. Once the other party finds the trace, the other party can infer its own intentions based on the intelligence search content. However, open source network intelligence work can be carried out under private conditions, making it impossible for the other party to detect and infer their intentions.

4. Importance of Intelligence Analysts

Jack Davis, a senior intelligence analysis expert of the Central Intelligence Agency, once clearly affirmed the importance of intelligence analysts, and proposed that 90% of US national security analysis is done by intelligence analysts. Steele, an American open source intelligence expert, has also put forward his own views on the process and working mode of open source intelligence. He believes that the most important skill of analysts in this mode is to "know who has the situation", and to be able to provide timely professional information to intelligence users who ask important questions.

Sources of intelligence from which qualified new knowledge is produced and quality-controlled by analysts. Passing ordinary information materials to ordinary users does not have much value. Only by analyzing and adding value on the basis of sufficient information materials, facing the problems that need to be solved, obtaining clear judgments, and forming reports that can be used for decision-making actions is Intelligence work was formed.

On the one hand, information retrieval cannot represent intelligence production in the current environment. "Intelligence" is not the information material itself, "intelligence" products are formed by analyzing and adding value on the basis of information materials, and highly rely on the personal intelligence of intelligence workers. Intelligence is not "found" directly, but "discovered" through analysis. Intelligent technologies such as big data crawlers, statistical analysis, and visualization can only play an auxiliary role in intelligence work, but cannot completely replace intelligence personnel. If the general information retrieval method fails to obtain intelligence from public information sources, and doubt the value of open source intelligence, it is a misunderstanding in understanding.

On the other hand, unlike traditional intelligence personnel, open source network intelligence analysts must be compound talents that adapt to the development of the new era. They must not only master the intelligence collection and analysis capabilities required by traditional intelligence work, but also need to master certain new technologies. With multiple languages and skills, it can work across regions and with multiple people.

5. Limitations of Open Source Cyber Intelligence Efforts

In the development of actual work, open source network intelligence work also has certain limitations and faces some difficult problems.

a. There is a lot of false information on the Internet.

Almost any information can be released through news, blogs, and social media sites, and many people regard news as a reliable source of information and believe that the veracity of news information can not be questioned. But "fake news" incidents emerge in endlessly, such as fake news about German refugees, fake news about ethnic conflicts in France, and fake news about South Korean politicians. According to research conducted by economics professors from Stanford University and New York University cited by the American Observer website, among the more than 700 news websites surveyed, fake news websites accounted for 64. Various sources of intelligence are numerous and complex, and additions, deletions, and modifications may occur in the links of provision, processing, release, and transmission, making it deviate from the original appearance. Especially in the era of mobile Internet, it is even more difficult to distinguish true and false information. At the same time, people may be affected by prejudice when distinguishing the authenticity of information. Therefore, in the face of massive amounts of information on the Internet, open source network intelligence analysts must be skeptical, objective, and fair to verify the information, analyze the authenticity of the information from different standpoints, and verify and restore the truth of the information from multiple data sources.

b. It is the problem of information overload.

According to statistics, in the global information system, the proportion of information waste is no less than 50%, and some disciplines even account for 80%. People have to spend a lot of time and energy to find the right information from massive amounts of information, so the cost of obtaining information is getting higher and higher. In addition, people are easily attracted by different texts, sounds or images, and other social media, and turn their attention to other things, get lost in the massive amount of information, and their focus and persistence on work have dropped significantly. Faced with the problem of information overload, analysts should follow the open source intelligence workflow in actual work. Analysts should follow the working cycle of clarifying mission requirements, doing mission planning and deployment, collecting data according to mission planning, using open source tools to process and analyze data, and reporting and summarizing, so as to avoid getting lost in massive data.

In general, open source network intelligence is widely applicable in various fields such as security and business. Countries around the world have begun to attach importance to the research and application of open source network intelligence. Establishing a stable, independent national open source intelligence agency is critical to open source intelligence work and research. A stable institution is conducive to absorbing open source intelligence practitioners and researchers from all sides, and an independent institution is conducive to avoiding the influence of the secrecy system of covert source intelligence agencies.