The Crowdotron-9000

Project

The goal of this project was to build a solution that could do live crowd detection with a mobile aerial vehicle. The system takes a live feed from an aerial platform, either a plane or a drone, and uses a neural network to perform object detection on the video.

These detections are then plotted as geographic coordinates using the vehicles telemetry to place the objects in the world. This data allows the system to perform detection, crowd tracking, and feedback control based on almost any object of interest. All of this is done live, which is much more challenging, and useful, than recording data for analysis later.

Method

The first part of the data pipeline is sourced from aerial footage streamed directly to a ground station alongside the telemetry of the vehicle. Both a plane and a quadrotor were built for this project, and while the plane is a better fit for operating time and range, the quadrotor is easier to collect stable data with. This data is then processed using a YOLO11 model trained on high angle security footage. The telemetry of the vehicle is then used to calculate the ground sample distance in order to get an approximation for where the objects that were detected in the camera frame are in relation to the vehicle.

Once this is done, the locations of the objects can be transformed into geographic coordinates using the vehicles latitude and longitude as provided by GPS. These object coordinates are constantly updated as new data is collected, allowing data analysis to be conducted live. This means that the control for the vehicle can be updated as detections are made, and that crowds and groupings can be evaluated as they form. The neural network used for this project is also very adaptable, and so long as the training data exists, this same system can be used to track people, cars, animals, or anything of interest.