Computers reconstruct 3D environments from 2D photos in a fraction of the time

Imagine trying to make an accurate three-dimensional model of a building using only pictures taken from different angles—but you’re not sure where or how far away all the cameras were. Our big human brains can fill in a lot of those details, but computers have a much harder time doing so.

This scenario is a well-known problem in computer vision and robot navigation systems. Robots, for instance, must take in lots of 2D information and make 3D point clouds—collections of data points in 3D space—in order to interpret a scene. But the mathematics involved in this process is challenging and error-prone, with many ways for the computer to incorrectly estimate distances. It’s also slow, because it forces the computer to create its 3D point cloud bit by bit.

Computer scientists at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) think they have a better method: A breakthrough algorithm that lets computers reconstruct high-quality 3D scenes from 2D images much more quickly than existing methods.

Their research is described in a paper, “Building Rome with Convex Optimization,” which recently received the Best Systems Paper Award in Memory of Seth Teller at the Robotics: Science and Systems Conference. It was authored by graduate student Haoyu Han and Heng Yang, assistant professor of electrical engineering at SEAS. The findings are published on the arXiv preprint server.

“By combining state-of-the-art AI depth prediction with a powerful new approach in convex numerical optimization, the method can estimate the positions of all points in a scene at once, with no need for step-by-step guesswork,” Han said. “As a result, the reconstruction process is not only faster and more robust than traditional techniques, but is also free from the need for initial guesses by the computer.”