The U.S. has been warning for weeks about the possibility of Russia invading Ukraine, and threatening retaliation if it does. Just eight years after Russia’s incursion into eastern Ukraine and invasion of Crimea, Russian forces are once again mobilizing along Ukraine’s borders.

As the U.S. and other NATO member governments monitor Russia’s activities and determine appropriate policy responses, the timely intelligence they rely on no longer comes solely from multimillion-dollar spy satellites and spies on the ground.

Social media, big data, smartphones, and low-cost satellites have taken center stage, and scraping Twitter has become as important as anything else in the intelligence analyst toolkit. These technologies have also allowed news organizations and armchair sleuths to follow the action and contribute analysis.

Governments still carry out sensitive intelligence-gathering operations with the help of extensive resources like the U.S. intelligence budget. But massive amounts of valuable information are publicly available, and not all of it is collected by governments. Satellites and drones are much cheaper than they were even a decade ago, allowing private companies to operate them, and nearly everyone has a smartphone with advanced photo and video capabilities.

As an intelligence and information operations scholar, I study how technology is producing massive amounts of intelligence data and helping sift out the valuable information.

Open-source intelligence

Through information captured by commercial companies and individuals, the realities of Russia’s military posturing are accessible to anyone via internet search or news feed. Commercial imaging companies are posting up-to-the-minute, geographically precise images of Russia’s military forces. Several news agencies are regularly monitoring and reporting on the situation. TikTok users are posting videos of Russian military equipment on rail cars allegedly on their way to augment forces already in position around Ukraine. And internet sleuths are tracking this flow of information.

Popular social media platforms like TikTok have become valuable sources of intelligence.

This democratization of intelligence collection in most cases is a boon for intelligence professionals. Government analysts are filling the need for intelligence assessments using information sourced from across the internet instead of primarily relying on classified systems or expensive sensors high in the sky or arrayed on the planet.

However, sifting through terabytes of publicly available data for relevant information is difficult. Knowing that much of the data could be intentionally manipulated to deceive complicates the task.

Enter the practice of open-source intelligence. The U.S. director of national intelligence defines Open-Source Intelligence, or OSINT, as the collection, evaluation, and analysis of publicly available information. The information sources include news reports, social media posts, YouTube videos, and satellite imagery from commercial satellite operators.

OSINT communities and government agencies have developed best practices for OSINT, and there are numerous free tools. Analysts can use the tools to develop network charts of, for example, criminal organizations by scouring publicly available financial records for criminal activity.

Private investigators are using OSINT methods to support law enforcement, corporate and government needs. Armchair sleuths have used OSINT to expose corruption and criminal activity to authorities. In short, the majority of intelligence needs can be met through OSINT.

Machine learning for intelligence

Even with OSINT best practices and tools, OSINT contributes to the information overload intelligence analysts have to contend with. The intelligence analyst is typically in a reactive mode trying to make sense of a constant stream of ambiguous raw data and information.

Machine learning, a set of techniques that allows computers to identify patterns in large amounts of data, is proving invaluable for processing OSINT information, particularly photos and videos. Computers are much faster at sifting through large datasets, so adopting machine learning tools and techniques to optimize the OSINT process is a necessity.

Identifying patterns makes it possible for computers to evaluate information for deception and credibility and predict future trends. For example, machine learning can be used to help determine whether information was produced by a human or by a bot or other computer program and whether a piece of data is authentic or fraudulent.

And while machine learning is by no means a crystal ball, it can be used – if it’s trained with the right data and has enough current information – to assess the probabilities of certain outcomes. No one is going to be able to use the combination of OSINT and machine learning to read Russian President Vladimir Putin’s mind, but the tools could help analysts assess how, for example, a Russian invasion of Ukraine might play out.

Technology has produced a flood of intelligence data, but technology is also making it easier to extract meaningful information from the data to help human intelligence analysts put together the big picture.

Written by Craig Nazareth, Assistant Professor of Practice of Intelligence & Information Operations, University of Arizona.

This article was first published in The Conversation.