The Science Inside Living Pictures

We’ve read some online comments and conversations from curious people who want to learn more about our light field picture (LFP) format. To help get answers, we did some research with our CTO, Kurt Akeley, to understand more about our team’s work on light field pictures.

A portion of the Stanford Multi-Camera Array that was used to create light field pictures in the early 2000s. (photo: Eric Cheng)

The leading question— why doesn’t the Lytro use an existing picture format? The team had to develop a unique picture format because light field pictures contain fundamentally different data than do traditional photographs, and because they use those data very differently to generate the images you see. The information is different because, while traditional cameras capture the intensity of the light, the sensor on our light field camera captures both the intensity and the direction of light. In total, 11 million rays (11 megarays) are captured, each describing the intensity of light along a path through the sensor. The information is used differently because megaray data are not viewed directly, but instead are projected from their (4-dimensional) ray space to a 2-dimensional image that you can view. As Kurt describes it, when you interact with a light field picture—for example, when you refocus it—”you aren’t changing the captured light field data, but are instead changing parameters that control projection of those data to the sequence of 2-D images that you see. Thus, light field pictures are ‘living pictures,’ and they make different demands of a picture format than do traditional photographs.”

Collecting and processing these different data presented an opportunity to rethink the picture format with specific goals in mind. Chief among these goals was to make it easy to share light field pictures online, despite the substantial megaray data involved. This goal is important because our company is focused on building cameras for the information age, which is all about online sharing and interaction. Sixty billion photos were shared on Facebook in 2010, and that number is expected to exceed 100 billion photos in 2011. However, the work required to achieve a standard of simple sharing is itself quite challenging.

Close-up view of a light field sensor that makes it possible for the Lytro camera to fit in your pocket.

One key was allowing living picture contents to be adapted depending on the requirements of the device on which they are viewed: the desktop, the web, or a mobile device. How does this work? When you shoot with a Lytro light field camera, each living picture includes the captured megaray data, along with public and private metadata that describe the circumstances of the picture. The Lytro desktop application receives these living pictures when you synchronize your camera with it. Because the megaray data are included, the desktop application can generate any of a wide range of projections of these data, including refocused images, or stereo image pairs for viewing on 3D displays.

Here’s the trick, though. When a living picture is shared, the viewer doesn’t have to download the original LFP. Instead, the megaray data are replaced with data that are optimized for viewing online or on a mobile device, dramatically reducing its size and simplifying the process of computing subsequent projections. (Of course, the megaray data remain in the original living picture stored by the desktop application.) Making sharing simple is the reason living pictures need a home “in the cloud”—it is the reason each Lytro camera purchase also gives you the ability to share and view your pictures on Lytro.com.

This is just the beginning of our work to explore the full potential of light field technology. For now, we’re very focused on getting the first Lytro camera in people’s hands. But, we are also thinking about developers. Besides supporting easy online sharing, another goal in designing our LFP format was to make it extensible to support future capabilities. Want to explore with us? Sign up for developer updates.

This entry was posted in Lytro Science and tagged Developer, Kurt Akeley, LFP, light field picture, living pictures,megaray, SDK by Lytro. Bookmark the permalink.

Lytro CameraThe Lytro camera has changed my life. It couldn't be any more modern.

If you don't know, the Lytro camera is the first [mass-produced] light-field camera.

From my jacket pocket, it only takes 1.2 seconds before it is ready to capture a 'LP' - 'Living Picture'. How long does it take for you to pull out your camera, turn it on, and have it be ready to actually capture an image? ...Unless you're holding one of these too, I doubt it's anywhere near 1.2 seconds... and you've missed that moment that you wanted to capture. That's no longer a problem, as long as you have your Lytro cam near you!

DON'T FORGET: This is only generation 1, so it is going to become infinitely smaller and faster (is that really possible?) in the future. Play around with the galleries below, and see for yourself. I heard through the grapevine @ CES that this technology will be implemented into cell phones by 2014. Enjoy!!! #ISM #aboutism.com #Lytro #Futech #Lytrography #IntegratedSocialMedia

Lytro is the brainchild of Dr. Ren Ng, a Stanford PhD whose dissertation on light-field technology five years ago was showered with awards. Now, with the help of $50 million in funding, most of it from Andreessen Horowitz, Ng has built a company that’s preparing to launch a focus-free digital camera later this year.

The basic premise of Lytro’s technology is pretty simple: The camera captures all the information it possibly can about the field of light in front of it. You then get a digital photo that is adjustable in an almost infinite number of ways. You can focus anywhere in the picture, change the light levels — and presuming you’re using a device with a 3-D ready screen — even create a picture you can tilt and shift in three dimensions. (I got a demonstration of the camera’s 3-D photos on a laptop and was blown away.)

You might think that this would produce unfeasibly large digital files, but Ng insists that the files will be roughly comparable to the average size of a digital photo today. The heavy lifting is being done by the camera’s on-board processors, he says. And because its light sensor is incredibly sensitive, you can capture low-light situations like restaurants a lot more easily — even without the flash.

You do not want to miss this: Although the camera itself isn’t due out until late 2011, Lytro on Tuesday unveiled a carousel of demonstration snapshots — all of them embeddable, available in Flash for the web and HTML5 for your smartphone. BE SURE TO CHECK THIS OUT! I already reserved mine!

INTEGRATED SOCIAL MEDIA INC.

Entries in lytro (5)

My Uncle Keng's Birthday Celebration in Living Pictures: