WHY THIS MATTERS IN BRIEF
- Cameras that can see around corners are getting better, and the technology is beginning more commercially viable but there’s still a long way to go before you can take pictures around corners
Building cameras that can see around corners is nothing new and over the past few years there have been a good number of attempts, by both MIT and the Chinese, but up until now none of those attempts have been able see around corners using just an ordinary smartphone setup, and while to many it might sound like something out of a science fiction movie that’s the idea behind a new algorithm out of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), and the discovery has implications for everything from emergency response to self-driving cars.
The CSAIL team’s imaging system, which can work with smartphone cameras, uses machine learning and algorithms to analyse information about the light reflecting back off of objects or people in a hidden scene in order to measure their speed and trajectory, all in real-time.
To explain, imagine that you’re walking down an L-shaped hallway and there are some objects around the corner. Whether or not you can see it those objects reflect a small amount of light on the ground in your line of sight, think of this as the difference between seeing a strong shadow, depending on the strength and direction of the light source, a weak shadow, or, taking it one step further, nothing visible at all – in the latter case just because you can’t see anything doesn’t mean that there isn’t any light being reflected on the floor in front of you, it’s just too weak for your eyes to sense. These “fuzzy shadows” are called “Penumbra.”
Using a video of the penumbra, CSAIL’s new smartphone system, which the team has dubbed “CornerCameras,” can stitch together a series of one-dimensional images that reveal information about the objects around the corner.
“Even though those objects aren’t actually visible to the camera, we can look at how their movements affect the penumbra to determine where they are and where they’re going,” says Katherine Bouman, who was lead author on a new paper about the system, “in this way, we show that walls and other obstructions with edges can be exploited as naturally-occurring ‘cameras’ that reveal the hidden scenes beyond them.”
Bouman says that the ability to see around obstructions would be useful for many tasks, from firefighters finding people in burning buildings to drivers detecting pedestrians in their blind spots.
Most approaches for seeing around obstacles involve special lasers. Specifically, researchers shine cameras on specific points that are visible to both the observable and hidden scene, and then measure how long it takes for the light to return. But, these so-called “time-of-flight cameras” are expensive and can easily get thrown off by ambient light, especially outdoors. In contrast, the CSAIL team’s technique doesn’t require actively projecting light into the space, and works in a wider range of indoor and outdoor environments and with off-the-shelf consumer cameras.
By viewing and analysing video of the penumbra, CornerCameras can produce a one dimensional image of the scene but 1D images aren’t especially helpful and can contain a lot of noisy data, but by observing the scene over several seconds the system can stitch together dozens of distinct images to determine an objects motion and trajectory.
“The notion to even try to achieve this is innovative in and of itself, but getting it to work in practice shows both creativity and adeptness,” says Professor Marc Christensen, who serves as dean of the Lyle School of Engineering at Southern Methodist University and was not involved in the research, “this work is a significant step in the broader attempt to develop revolutionary imaging capabilities that are not limited to line-of-sight observation.”
The team was surprised to find that CornerCameras worked in a range of challenging situations, including weather conditions like rain.
“Given that the rain was literally changing the color of the ground, I figured that there was no way we’d be able to see subtle differences in light on the order of a tenth of a percent,” says Bouman, “but because the system integrates so much information across dozens of images, the effect of the raindrops averages out, and so you can see the movement of the objects even in the middle of all that activity.”
The system still has some limitations. For obvious reasons, it doesn’t work if there’s no light in the scene, and can have issues if there’s low light in the hidden scene itself. It also can get tripped up if light conditions change, like if the scene is outdoors and clouds are constantly moving across the sun, and with smartphone quality cameras the signal also gets weaker as you get farther away from the corner.
The researchers plan to address some of these challenges in future papers, and will also try to get it to work while in motion. The team will soon be testing it on a wheelchair, with the goal of eventually adapting it for cars and other vehicles.
“If a little kid darts into the street, a driver might not be able to react in time,” says Bouman, “while we’re not there yet, a technology like this could one day be used to give drivers a few seconds of warning time and help in a lot of life-or-death situations.”
The work was supported, in part, by the DARPA REVEAL Program and will no doubt one day make its way out of the lab and into the real world.
