Today begins the third annual IndieCade East at the Museum of the Moving Image, consisting of three days of talks, panels, workshops, exhibits, and games, all curated from IndieCade’s diverse and brilliant community of designers, thinkers, and players. This year's IndieCade East features a special exhibit of games for attendees to play, including a showcase of virtual reality games generated as part of tech innovator Leap Motion's 3D Jam. In addition, Leap Motion will also be hosting an IndieCade workshop titled "Tips from the Metaverse: Building VR Experiences with Motion Control" on Saturday, February 14th. Sloan Science and Film spoke with Daniel Plemmons, interaction designer and front-end engineer at Leap Motion, about the science behind virtual reality and what this means for the future of the gaming industry.

Sloan Science and Film: Can you give an overview of your role at Leap Motion?

Daniel Plemmons: I'm a designer and software engineer on what we call our front-end team. My job is to create new designs for interfaces and experiences using the Leap Motion Controller, and then share what we've learned with the developer community. We build a lot of tools and examples for developers to work from, and give feedback to our hardware and tracking teams to help them focus their work on the features most useful to developers. Most recently we've released a super cool VR Planetarium demo and an updated version of our VR widgets UI assets.

SSF: What brought you to virtual reality game engineering?

DP: I studied interaction design and game development at the Savannah College of Art and Design's Atlanta campus. While I was there, my professors got me interested in natural user interfaces and how the methods we use to interact with computers have a foundational impact on our perception of those interactions. VR offers a whole new exciting realm of exploration. Motion control in VR lets you engage your body in interactions in meaningful ways, while providing all kinds of great sensory feedback. There's an unexplored design space there that I'm really excited to dive into. I'm particularly interested in pulling from my past work and looking at how we can combine tangible interfaces with optical motion tracking and VR to create experiences that wouldn't have been possible just 2-3 years ago.

SSF: Describe the science behind virtual reality - how does it work and how was it developed?

DP: A lot of different technologies come together to make a great VR experience possible, but the real linchpin is your stereoscopic vision. When you look around at the world, your brain uses the subtle differences in the images coming from your left and right eyes to determine how close or far away objects in your field of view are. Head mounted displays, or HMDs, split their screen and use some crafty distortion tricks and lenses to display slightly different images to your eyes to mimic what you would see in the real world. Combine this with gyroscopes and accelerometers (like in your phone) to track your motion so the virtual cameras move with your head, and you’ve got a VR headset. Our technology uses the same stereoscopic vision principal to provide motion control input. Our tracking software uses the differences between two infrared sensors (among other data) to determine the distance of objects in the field of view.

Most of the technology that powers HMDs has been around for many years, but only now are costs, computing power, and the market all aligned to make virtual reality a viable product.

SSF: What does it do to your perception?

DP: Modern VR headsets do a great job of blocking out the world around you and keeping your attention in the virtual space. Between the headset tracking your motion, the excellent illusion of depth, good audio, and natural input enabled by motion control technology like ours, you can very easily slip into the sense of actually being in a virtual space.

Even when you're standing still in the real world, virtual reality can make you feel like you are poking your head up into a whole new world inside the headset. An interesting VR illusion can be seen in our Planetarium demo, which features a digital bracelet that displays useful information on the virtual projection of your arm. A number of people said they could "feel" the bracelet even though it only existed in the virtual space.

SSF: What are the hurdles in using it for gaming purposes? What does it allow that traditional consoles do not?

DP: To my mind, the biggest hurdles for gaming in the near term are going be the power of most people's home computers and the isolating nature of VR. You need very high frame rates in an HMD to avoid causing motion sickness; the computing power it takes to run modern games at those speeds is pretty immense. Those costs will come down in time, but for now there’s a chance that some people will have poor experiences trying to run games on underpowered computers.

The second issue is a bit more ephemeral. Since VR headsets are opaque, you lose your awareness of the world around you. It can be isolating and if there are others around, it can be a bit strange in a social context. This may become more normal as time goes on, similar to when Bluetooth headsets first appeared. Things like video pass-through (enabled through our Raw Image API – you see what the Leap Motion’s cameras see) also help to merge the real world with the virtual world. That said, I think there will be a period where we will need to get used to people around us sometimes having headsets on.

In terms of what HMDs give you that traditional consoles can’t, it really comes down to that sense of presence and the freedom to move and look around naturally.

SSF: Can you describe the games that will be showing at IndieCade?

DP: The games we’re showing at IndieCade are finalists from the Leap Motion 3D Game Jam we ran in cooperation with IndieCade. There’s an awesome breadth of work, from the contemplative nod to classic science fiction, Weightless, to the education-focused World of Comenius that explores new ways to learn about the world through virtual reality. We’re also showing some of the work the Leap Motion front-end team has been creating. You can find out more about each of the finalists' games here.

SSF: Do you see VR replacing more traditional gaming systems?

DP: Games are a huge and varied space where different platforms lend themselves to different experiences. There will be plenty of games that are always best on a TV or other platform like mobile phones. Couch multiplayer games like Towerfall or Halo come to mind. But video-gaming, especially AAA gaming, has been on a trajectory of pushing the realism and immersion boundaries of technology since its inception with Spacewar way back on the PDP-1's oscilloscope display. For games chasing that sense of realism, immersion, and presence, I can't think of a better modern platform than virtual reality.

SSF: Where does it stand in terms of being a true consumer item? What can we expect from it in the future?

DP: We're right on the cusp of VR being a real product and a real force in the entertainment industry. Already plenty of non-developers are buying up Oculus development kits intending to play with the demos and experiences already in production from tons of developers. On top of that, a bunch of new VR headsets, input devices, and accessories were announced at CES this year. You may have seen that we are one of the principal members of the OSVR (open source virtual reality) initiative; we introduced support via an official plugin during CES. There are a lot of big players investing time and money into the space. How soon VR will become mainstream remains to be seen, but I figure we'll see some of the first products really take off this upcoming holiday season.

SSF: What are the possibilities for the technology outside the gaming community in real-world scenarios?

DP: Games are a great sandbox for new computing technology, but it’s hardly where VR will end. There are already a lot of folks thinking about the various applications for VR technology. The perspectives afforded by VR have applications in the medical imaging and diagnostic fields, industrial work, training and simulation, telepresence, data-visualization and a plethora of other fields. Coming out of the arts world, I am really excited to see how directors and videographers adapt film techniques to virtual reality, like what was seen at the Sundance Film Festival. It’s a really great time to be working with natural interface technology.