The Science Inside Living Pictures:

The Science Inside Living Pictures

We’ve read some online comments and conversations from curious people who want to learn more about our light field picture (LFP) format. To help get answers, we did some research with our CTO, Kurt Akeley, to understand more about our team’s work on light field pictures.

A portion of the Stanford Multi-Camera Array that was used to create light field pictures in the early 2000s. (photo: Eric Cheng)

The leading question— why doesn’t the Lytro use an existing picture format? The team had to develop a unique picture format because light field pictures contain fundamentally different data than do traditional photographs, and because they use those data very differently to generate the images you see. The information is different because, while traditional cameras capture the intensity of the light, the sensor on our light field camera captures both the intensity and the direction of light. In total, 11 million rays (11 megarays) are captured, each describing the intensity of light along a path through the sensor. The information is used differently because megaray data are not viewed directly, but instead are projected from their (4-dimensional) ray space to a 2-dimensional image that you can view.  As Kurt describes it, when you interact with a light field picture—for example, when you refocus it—”you aren’t changing the captured light field data, but are instead changing parameters that control projection of those data to the sequence of 2-D images that you see. Thus, light field pictures are ‘living pictures,’ and they make different demands of a picture format than do traditional photographs.”

Collecting and processing these different data presented an opportunity to rethink the picture format with specific goals in mind. Chief among these goals was to make it easy to share light field pictures online, despite the substantial megaray data involved. This goal is important because our company is focused on building cameras for the information age, which is all about online sharing and interaction. Sixty billion photos were shared on Facebook in 2010, and that number is expected to exceed 100 billion photos in 2011. However, the work required to achieve a standard of simple sharing is itself quite challenging.

Close-up view of a light field sensor that makes it possible for the Lytro camera to fit in your pocket.

One key was allowing living picture contents to be adapted depending on the requirements of the device on which they are viewed: the desktop, the web, or a mobile device. How does this work? When you shoot with a Lytro light field camera, each living picture includes the captured megaray data, along with public and private metadata that describe the circumstances of the picture. The Lytro desktop application receives these living pictures when you synchronize your camera with it. Because the megaray data are included, the desktop application can generate any of a wide range of projections of these data, including refocused images, or stereo image pairs for viewing on 3D displays.

Here’s the trick, though. When a living picture is shared, the viewer doesn’t have to download the original LFP. Instead, the megaray data are replaced with data that are optimized for viewing online or on a mobile device, dramatically reducing its size and simplifying the process of computing subsequent projections. (Of course, the megaray data remain in the original living picture stored by the desktop application.) Making sharing simple is the reason living pictures need a home “in the cloud”—it is the reason each Lytro camera purchase also gives you the ability to share and view your pictures on Lytro.com.

This is just the beginning of our work to explore the full potential of light field technology. For now, we’re very focused on getting the first Lytro camera in people’s hands. But, we are also thinking about developers. Besides supporting easy online sharing, another goal in designing our LFP format was to make it extensible to support future capabilities. Want to explore with us? Sign up for developer updates.

This entry was posted in Lytro Science and tagged Developer, Kurt Akeley, LFP, light field picture, living pictures,megaray, SDK by Lytro. Bookmark the permalink.