A Monte Carlo tradeoff analysis to guide resource investment in threat detection systems: From forensic to prospective investigations

The Department of Defense has invested in threat detection systems, consisting of sensors and algorithms to collect and process data on potential threats. We performed a Monte Carlo tradeoff analysis to determine the optimal combination of sensor versus algorithm capability for an airborne wide area motion imagery (WAMI) system used to identify insurgents detonating improvised explosive devices (IEDs) in urban environments. We created a computational model of the WAMI system, including the camera and algorithms for processing, exploitation, and dissemination. We also synthesized input data consisting of objects and activities taking place in a city over two days. We compared the model’s output versus input to quantify its overall value to the counter-IED mission. We exercised the model twice: forensically to identify insurgents after the IED explosions and prospectively to identify insurgents as early as possible beforehand. Although better sensors led to better intermediate results, they did not always lead to greater overall value. Once the number of true positives saturated at its best possible value, the number of false positives continued to grow as the quality of the sensor improved. Similar results may occur for other scenarios in which threats are limited in number with low prevalence.