Defense Intelligence Agency Expected to Lead Military’s Use of ‘Open Source’ Data

DIA director Lt. Gen. Scott Berrier told a Defense Department technology conference in Phoenix this week that the agency was expecting to get a directive from Pentagon leadership to designate the agency the manager of open-source intelligence—often called OSINT—for all military intelligence programs.

“There’s some controversy with OSINT right now—the tools that we’re using, the tradecraft that we’re practicing, the amount of money that we’re paying for data, who is paying what and what’s being duplicated, how does [publicly available information] fit into all of that,” Gen. Berrier said in remarks before the conference in Phoenix this week. The directive by the Pentagon’s undersecretary of defense for intelligence is expected “very, very soon,” Gen. Berrier added.

A spokesman for the Pentagon declined to comment on “pre-decisional” matters, including whether there was an expected timeline for the action.

Some people within the intelligence community have proposed creating a new agency specifically dedicated to open-source information. The expected Defense Department action would stop short of that and doesn’t address many thorny privacy and policy questions about how much information governments should be able to obtain on citizens.

At the conference, DIA officials said about 80% of what is in agency intelligence reports now is from unclassified sources. They emphasized that the military was ingesting so much data—much of it from unclassified open sources—that processing and analytical power were now in short supply.

“People used to ask me, what additional information do your analysts need? They’ve got lots of information. They need the ability to process it. They need the ability to understand it,” Gregory Ryckman, the DIA’s deputy director for global integration, told attendees of the Department of Defense Intelligence Information System Worldwide Conference.

Open-source intelligence once referred mostly to monitoring press reports from around the globe. The U.S. government has had offices responsible for monitoring foreign media outlets since the 1940s and expanded into monitoring foreign scientific and technical journals during the Cold War.

In recent years, OSINT has come to encompass an array of unclassified material available online or for purchase. That includes social media and posts on web forums as well as commercially available data collected by technology giants, advertising entities and consumer-facing companies. It also includes a huge category of data that modern electronic devices emit and that are often collected without much consumer awareness, such as precise geolocation, IP addresses, wireless identifiers associated with cars and modern Bluetooth devices.

The government has also invested heavily in developing specialized analytical capabilities to extract information from open sources—for example, tools that can estimate location information from social-media photos using tiny clues such as the position of the sun, stars or visible landmarks like mountains.

Depending on the intelligence priority, these data sets can now be equally or more valuable to government agencies than more traditional, intrusive types of government surveillance like wiretaps, some former U.S. intelligence officials have said. Unlike such sources, open-source retrieval doesn’t require approval from the Foreign Intelligence Surveillance Court.

Open-source intelligence has played a significant role in military operations in recent years, especially in the global campaigns against al Qaeda and Islamic State, according to people familiar with the matter and documents viewed by The Wall Street Journal. Some of the lethal airstrike targeting in the campaign against Islamic State was done in part through social media—including by identifying and killing terrorist targets who made critical operational security errors like leaving GPS coordinates attached to public social-media posts, the people say.

Foreign governments have come to rely on open-source intelligence, too. The pervasiveness of what some in the field call “digital dust” in modern society has complicated efforts of U.S. spies to remain undercover in overseas postings, according to current and former officials.

The government’s expanding use of open-source data has recently begun to attract significant concern from privacy advocates. The Privacy and Civil Liberties Oversight Board, a small federal agency tasked with reviewing the government’s surveillance practices, has said it is reviewing the Federal Bureau of Investigation’s use of commercially available open-source data sets in relation to its counterterrorism efforts.

The DIA also revealed earlier this year that it had access to data on the movement of millions of American smartphones through data drawn from apps and that it had queried the data five times without warrants in the past few years. All five instances involved matters of national security, the Journal has reported, citing a person familiar with the matter.

“DIA has policies and procedures in place to ensure compliance with all Constitutional, statutory and regulatory requirements and attorney general-approved guidelines for the conduct of intelligence activities,” a DIA spokeswoman said about the agency’s use of OSINT.

The DIA, based in Washington and created 60 years ago, is one of only a handful of intelligence agencies tasked with analyzing information from any information stream, whether gleaned from human spies, cyber data or satellite imagery. In recent years some lawmakers have unsuccessfully proposed overhauling DIA, expressing concerns that it was too large, lacked a core mission focus and was duplicating work of other agencies.

An influential report released early this year by the Center for Strategic and International Studies’ Technology and Intelligence Task Force recommended establishing a new agency dedicated to OSINT and called for more investment in OSINT techniques and training.

Glenn Gerstell, a former general counsel of the National Security Agency, said the DIA was a natural choice for the Pentagon to take a formal lead in the area.

“Open-source information is going to be of incalculable value both to our intelligence community and military in the future,” he said.

Write to Byron Tau at [email protected] and Dustin Volz at [email protected]