A Guide to Open Source Intelligence Gathering (OSINT)
Immeasurable amounts of personal and potentially incriminating data are currently stored in the websites, apps, and social media platforms that people access and update daily via their devices. Those data can become evidence for citizens, governments, and businesses to use in solving real financial, employment, and criminal issues with the help of a professional information gatherer.
Many people think using their favorite Internet search engine is sufficient to find the data they need and do not realize that most of the Internet is not indexed by search engines. SEC487 teaches students legitimate and effective ways to find, gather, and analyze this data from the Internet. You'll learn about reliable places to harvest data using manual and automated methods and tools. Once you have the information, we'll show you how to ensure that it is sound, how to analyze what you've gathered, and how to make it is useful to your investigations.
Passive Reconnaissance or OSINT or Information Gathering is to gather information about a target using publicly available information. Extracting relevant information about the target always helps and plays a significant role during bug bounties. The primary tasks during reconnaissance include:
Identification of IP addresses, subdomains, ports and services that can increase our attack surface.
Identification of technologies used, application platform and other infrastructure details
Identification of sensitive information for e.g. API keys, AWS S3 buckets, leaked credentials, etc.
Other data includes identification of Log files, Backup files, Database files, Client-side code, Javascript libraries and Configuration files
Usually one of the primary step during Reconnaissance is to identify the net ranges and sub-domains associated with the target. Search engines like Shodan and Censys can be used to a great deal for finding IP addresses, sensitive files, server information, error messages and other exposed services.
A whois lookup helps to find admin contact and other email addresses. Email addresses can be searched through database leaks or through a search service like HaveIBeenPwned that tells you if your email was found as part of a breach.
This whois site https://ipalyzer.com/ is extremely useful. You can get tons of information about the target like GeoIP information, provider information and can query different spam lists.
Wappalyzer is also a good utility to uncover technologies used on websites. It detects content management systems, ecommerce platforms, web frameworks, server software, analytics tools and many more. Analyzing HTTP status codes and response headers using burp is a must. Check for CSP headers as they contain domain names and sources from where the script loading may be allowed.
One should always look for code hosting services like github, gitlab, bitbucket etc. Web vulnerabilities, configuration issues, AWS and other secret keys can sometimes be found in these services. While making commits, developers leave credentials or unique “Access tokens”. It’s easy for attackers to retrieve this sensitive information by using commit logs and checking out specific commits and host an attack on the target infrastructure.
Tools like Github Dorks can be used to to search sensitive data like private keys, credentials, authentication tokens etc in the github repositories.
In the field of cybersecurity, using the right tools for your OSINT investigation can be really effective if you combine it with critical thinking and have a clear OSINT strategy. Whether you are running a cybersecurity investigation against a company/person or if you are on the opposite side working to identify and mitigate future threats, having pre-defined OSINT techniques and clear goals can save you a lot of time. Every organisation should embrace OSINT as one of the cybersecurity defenses to identify and detect app, services, and/or server threats.