FlockAI: A Testing Suite for ML-Driven Drone Applications

Machine Learning (ML) is now becoming a key driver empowering the next generation of drone technology and extending its reach to applications never envisioned before. Examples include precision agriculture, crowd detection, and even aerial supply transportation. Testing drone projects before actual deployment is usually performed via robotic simulators. However, extending testing to include the assessment of on-board ML algorithms is a daunting task. ML practitioners are now required to dedicate vast amounts of time for the development and configuration of the benchmarking infrastructure through a mixture of use-cases coded over the simulator to evaluate various key performance indicators. These indicators extend well beyond the accuracy of the ML algorithm and must capture drone-relevant data including flight performance, resource utilization, communication overhead and energy consumption. As most ML practitioners are not accustomed with all these demanding requirements, the evaluation of ML-driven drone applications can lead to sub-optimal, costly, and error-prone deployments. In this article we introduce FlockAI, an open and modular by design framework supporting ML practitioners with the rapid deployment and repeatable testing of ML-driven drone applications over the Webots simulator. To show the wide applicability of rapid testing with FlockAI, we introduce a proof-of-concept use-case encompassing different scenarios, ML algorithms and KPIs for pinpointing crowded areas in an urban environment.