U.S. Spies Want Algorithms to Spot Hot Trends



The U.S. intelligence community wants a sharp competitive edge on the world’s best and brightest ideas. In an effort to find the next big thing before it happens, they’re looking to do away with fallible human trendspotters, and enlist an algorithmic system to “scan the horizon” and tap into the first signs of burgeoning memes in science and technology.

Iarpa, the intel world’s far-out research arm, is already wary of trusting big calls and predictions to flesh-and-blood experts alone. Earlier this year, the agency solicited proposals for a system that would evaluate and rank the value of expert opinion based on niche, learning style, prior performance and “other attributes predictive of accuracy.”



This time around, Iarpa’s looking for a system that wouldn’t just rate experts, but would take over many of their responsibilities entirely. The agency’s Foresight and Understanding from Scientific Exposition (or FUSE) wants researchers to create “a reliable, evidence-based capability that…reduce[s] the labor involved to identify specific technical areas for in-depth review.”


As Iarpa’s solicitation notes, trying to identify the hottest trends before they heat up is time-consuming, time sensitive and susceptible to human bias. Not to mention that most experts are confined to certain geographic regions, cultures, languages and technical niches. But with globalization churning out innovations worldwide, Iarpa wants a system that can operate in several languages and account for cultural differences.

The end-product that Iarpa’s after would start by poring over sets of loosely related documents, like studies, patents and government reports, on topics as specific as facial recognition or as broad as artificial intelligence. Algorithms would then weigh factors like key words, publication dates and locations, pin down trends over time, and come up with a series of statements and rankings to provide “compelling evidence” that a given innovation or idea will either emerge or fizzle.

Not only do proposers need to develop a system that can operate in six languages (English, German, Japanese, Chinese, Russian, Spanish, and Korean), but they need to start by creating a “theory of emergence” upon which the system’s algorithms will be based. For example, how to weigh the value of increased government funding versus a surge in patent documents.

In other words, while Iarpa wants an code-driven prediction system, it’s one that’ll inevitably be founded on fallible human input. LINK