Putting AI vision into better focus with method that mimics human processing

Today’s AI vision is effective at recognizing simple images in isolation—such as buildings, cars, and people. But when it’s called upon to identify more complex terrain, its accuracy becomes questionable. This is one of the challenges facing self-driving car technology. AI visual systems must correctly spot buildings, cars, and people—all at the same time and in a fluid environment, such as a busy intersection.

“Can we develop a learning algorithm that can directly handle data coming from what we experience—as opposed to merely recognizing simple images on a computer screen?” asks Mengye Ren, an assistant professor at NYU’s Courant Institute of Mathematical Sciences and Center for Data Science.

Ren and his colleagues are building an algorithm that would do just that, enabling AI systems to learn from their environment—a street, an ocean, or even another planet—in order to effectively identify its surroundings.

Their method, PooDLe, is inspired by how humans and animals process cluttered scenes. It captures both foreground images (e.g., pedestrians crossing the street) and background images (distant cross streets) using “optical flow”—information about how pixels move between video frames. This process allows for the identification of paired regions containing the same object across time—such as a pedestrian moving from the curb to a crosswalk and continuing down a crowded street.

“PooDLe combines the best of existing AI vision tools by recognizing both big and small objects,” explains Mengye Ren, an assistant professor at NYU’s Courant Institute of Mathematical Sciences and Center for Data Science. “Our goal is to continue to enhance this tool so it can perceive various objects in a scene—cars, roads, traffic lights, cyclists, and so on.”