How to enhance OSINT investigations using AI

What is OSINT?

Investigators can gather OSINT from publicly accessible online and offline sources. These include the following:

• Internet resources, which can be freely available or purchasable. Examples include social media posts, videos and images on media-sharing websites, discussion forums, blogs, and public databases such as vital records.
• Traditional media, such as newspapers, magazines, television and radio broadcasts, and billboards.
• Gray literature, consisting of academic materials like books, journals, theses and dissertations.
• Business records, including corporate filings, product catalogs, patents and other business documentation.

Who benefits from using OSINT?

OSINT is highly versatile; it serves the needs of various user groups across industries. Common OSINT users include the following:

• Governments. OSINT can support national security efforts by helping government bodies identify potential threats and track geopolitical developments. Officials can also use OSINT to monitor and predict public sentiment, helping them evaluate policy acceptance or attitudes during a national crisis.
• Security services and intelligence agencies. OSINT can help combat cybercrime and terrorism by uncovering malicious activities, tracking threat actors and identifying vulnerabilities on a national scale. Security services also use OSINT to identify propaganda or misinformation campaigns on social media.
• Financial institutions. Banks and financial services use OSINT to ensure customer and supplier due diligence and compliance with regulatory requirements.
• Corporations. Businesses use OSINT for market research, consumer behavior analysis and competitive intelligence. For example, analyzing social media trends can help companies tailor products to specific demographics or regions.
• Investigative journalism and human rights. OSINT helps journalists and activists uncover hidden stories, expose human rights violations and verify sources' claims.
• Law firms and private investigators. OSINT can assist in finding evidence, locating individuals and conducting due diligence in legal cases, such as fraud or intellectual property theft lawsuits.
• Cyberthreat intelligence analysts and penetration testers. OSINT enables analysts to identify security threats and understand attacker techniques. Similarly, penetration testers use OSINT during reconnaissance to gather information about target organizations, which helps them simulate real-world attacks.

Using AI to support OSINT investigations

OSINT investigations require processing vast amounts of digital data from diverse public sources. AI-powered tools can improve efficiency and accuracy at several stages of this process.

1. Automate data collection

Traditional web scrapers struggle with unstructured data formats, such as Microsoft Office files, PDFs, XML, CSV files, images and videos. In contrast, AI scrapers excel in handling such data, enabling OSINT investigators to save time on data collection and obtain more contextually relevant information.

AI-powered web scrapers automatically harvest public data from social media platforms, news websites, blogs, discussion forums, web archives and public databases. They can adapt to website layout variations and extract only relevant data for investigation. They are also helpful in identifying specific keywords in scanned documents, extracting metadata from multimedia files such as images and videos, and recognizing named entities across large data sets.

There are many AI-powered web scrapers on the market. Popular options include the following:

• AnyPicker, a free web scraper installed as a Chrome extension.
• Browse AI, a paid tool with preconfigured bots for data-harvesting use cases. A free version is also available.
• ParseHub, a free web scraper that screens webpages and understands element hierarchy.

2. Analyze large data sets

AI tools can filter and categorize extracted data based on predefined criteria such as keywords, sentiment or specific entity names. This enables OSINT gatherers to focus on the most relevant information for their investigations.

For example, in geopolitical analysis, an OSINT gatherer monitoring social media sites for political unrest can use AI tools to extract and categorize public posts containing keywords like protests, demonstrations, strikes and boycotts. The tool can further analyze sentiment to identify posts expressing negative emotions or dissatisfaction. This helps the analyst prioritize regions or groups that require scrutiny.

AI can also analyze discussion forums and dark web marketplaces to find posts that mention specific keywords, such as zero-day exploit, ransomware and stolen credentials. By categorizing mentions based on threat type or associated entities, OSINT analysts can identify emerging cybersecurity risks.

Some web scrapers offer functionalities to analyze collected data. But there are also dedicated AI tools for OSINT data analysis, including the following:

• Ovis, which identifies objects in image and video files.
• Lenso.ai, an AI-powered face search engine.
• Dataminr, which detects threats and responds to emerging risks to individual, brand, physical and virtual assets.
• Paliscope, a comprehensive OSINT platform for searching, visualizing and exploring data.

3. Summarize data

Natural language processing (NLP) tools can summarize large amounts of textual data from various sources. This enables OSINT gatherers to extract key insights from long-form content, such as articles, reports and social media posts, without manual review.

For example, NLP can summarize financial reports and audit findings during a corporate investigation. OSINT analysts investigating a company might use AI to extract and summarize sections mentioning abnormal transactions or compliance violations, saving time and effort.

NLP-powered tools can also identify and classify named entities such as people, organizations, locations and dates within large data sets. For instance, when analyzing thousands of social media posts during a political crisis, AI can highlight mentions of particular leaders, organizations, political parties or regions.

4. Perform predictive analytics

AI tools can conduct predictive analytics to anticipate future trends, behaviors and risks. AI tools can facilitate predictive analysis in the following ways.

Trend identification and forecasting. AI can process large data sets from various sources to identify emerging trends. For example, to predict potential market downturns or political unrest in a particular country, investigators could use AI to detect rising mentions of layoffs, inflation or supply chain disruptions.

Social media sentiment analysis. AI tools can predict public responses to events, policies or products by analyzing social media sentiment. For instance, during an election, AI can analyze millions of Facebook posts to assess voter sentiment, helping predict shifts in public opinion or protests in specific regions.

Tools that can perform social media sentiment analysis include the following:

• Free Sentiment Analyzer, which performs sentiment analysis on English-language text.
• FaceReader, which analyzes facial expressions.
• Social Media Sentiment Visualization, which performs text sentiment analysis on social media platforms.

Nihad A. Hassan is an independent cybersecurity consultant, expert in digital forensics and cyber OSINT, blogger, and book author. Hassan has been actively researching various areas of information security for more than 15 years and has developed numerous cybersecurity education courses and technical guides.