Penn Engineers Speed Up Event Camera Vision to 440 Times per Second

Predicting motion instead of isolated tasks

Standard cameras record complete visual scenes at regular intervals, but event cameras operate on a different principle. Each individual pixel registers changes in light intensity continuously, capturing rapid movement without assembling traditional frames, as reported by Interesting Engineering.

Because these sensors emit millions of scattered signals rather than structured frames, conventional computer software struggles to interpret them without separate programs for object detection, distance measurement, or motion tracking. The Penn team solved this bottleneck with a neural network called Fast Feature Field (F³). Instead of training separate models for distinct visual tasks, the researchers trained the system to predict upcoming signals based on past data.

"To accurately anticipate future events based on past ones, the network must learn to represent both the structure of a scene and the motion taking place within it," explained first author Richeek Das in findings detailed by Interesting Engineering. This predictive model extracts geometry and motion in a single processing step.

“To accurately anticipate future events based on past ones, the network must learn to represent both the structure of a scene and the motion taking place within it”

— Richeek Das, lead author, University of Pennsylvania

“Our goal is to make using an event camera as straightforward as using a conventional one”

— Pratik Chaudhari, associate professor, University of Pennsylvania

Skipping 95 percent of empty visual points

Event sensors generate tens of millions of discrete signals every second, which easily overwhelms mobile computing chips. However, the researchers observed that roughly 95% of possible sensor coordinates register no activity during a given instant.

F³ ignores these inactive positions completely and focuses calculations solely on coordinates where light shifts occur, according to Interesting Engineering.

This filtering method cut down unnecessary calculations. When processing high-definition event data, F³ runs at 120 times per second. At lower VGA resolution, its speed increases to 440 times per second, enabling real-time robotic perception.

Tests on quadruped robots and night streets

The researchers evaluated their system using sensor feeds gathered from cars, four-legged robots, and autonomous flying drones. Experimental runs spanned difficult visual conditions, including dark urban streets at night and crowded indoor corridors, according to Interesting Engineering.

The software matched or exceeded existing robotic vision benchmarks while running two to five times faster. It also lets roboticists adapt existing vision tools to event data, removing the need to design custom software for each deployment.

While the publication does not list a formal patent application number, the technical architecture represents an intellectual property asset developed through academic research that could bridge experimental sensors and practical robotics.

Direct hardware integration for autonomous drones

The Penn research group is currently installing F³ aboard autonomous aerial drones designed to fly through cluttered, dark spaces.

The long-term plan targets the sensor hardware itself, according to Interesting Engineering. The engineers intend to embed the algorithmic model directly into camera chips, transforming raw electrical impulses into usable visual data on board.

"Our goal is to make using an event camera as straightforward as using a conventional one," stated Pratik Chaudhari, associate professor at the University of Pennsylvania, in comments reported by Interesting Engineering. The team published their complete technical evaluation in the journal Science Robotics.

Read also on The Patent: Trilateral patent offices meet in Vienna on Global Assignment and AI · MIT engineers build a snake-inspired soft robot to explore the small intestine