There’s an oft-cited quote of Walt Disney’s: “It’s kind of fun to do the impossible.” Were he alive today, it would be interesting to see what he’d make of – and with – modern virtual reality (VR) systems.

All VR requires a basic core functionality: The ability to deliver an environment that matches perception. In the simplest VR, a view is presented to the user. If they look down, they see an image of the floor. Look up, they see sky. Done quickly and accurately, the user has a subconscious reaction to the simulation, often referred to as “presence.” Done too slowly, or inaccurately, the illusion is broken. Worse still, the user may be made uncomfortable, or even nauseated.

A few months ago, Podcaster Kent Bye outlined an “Elemental Theory of Presence” that described several types of presence: Emotional, Social and Mental, Embodied, and Active. Emotional and Social and Mental Presence are higher-order functions tied to content, context, and usages.

Embodied Presence refers to the basic scenario I sketched out above, which is to fool the brain into thinking you are physically located in another place. This is a starting point for VR but insufficient for true immersion. Embodied Presence creates the illusion of VR. Active Presence sustains it.

Active Presence comprises system elements that sense users’ actions and use that input to affect the simulation. “Sensing user activity” covers a wide range of elements: 3- and 6-degrees-of-freedom movement, input peripherals (e.g. gamepads, motion-tracked controllers), environment-sensing cameras, voice input, etc. “Affecting the simulation” refers to the degree to which the simulation has been designed to recognize these activities and respond.

Controllers and Affordance

As with Embodied Presence, Active Presence today is only just crossing a minimum threshold of believability. Specifically, we are seeing a great deal of progress in controller capabilities and affordance.

Recently, VR has seen a rapid increase in controller capabilities. We’ve gone beyond simple gamepads with buttons mapped to actions or directions. Valve and HTC set a bar for the industry with directionally tracked controllers that is quickly becoming a minimum requirement for truly immersive VR. Oculus’ Touch added finger and thumb sensors to detect those digits leaving the controller, allowing some increased gestures and actions. Now Valve is shipping early units of its “Knuckles” controller, expanding the idea of finger sensing to all five digits, allowing a wider range of gestures and actions.

The goal of expanding the capabilities of these controllers is not to put more sensors in them for more accuracy or precision (though that’s certainly a benefit). Rather, these capabilities expand controller affordance – allowing them to map to a wider range of experiences, a wider set of possible actions.

For example, the Vive controllers map very well to things like holding a tennis racquet, or a pistol. However they map poorly to things like mimicking hands grasping things. For example, while Owlchemy Labs delivered a fantastic experience in Job Simulator, despite the limited affordance of today’s controllers, it does limit immersion when the user has to learn how to do something that should come naturally.

Interaction designer Karen McClure refers to the concept of “fitting the person to the machine, versus fitting the machine to the person.” Applying this concept in VR, we need controllers that allow the user to interact with the virtual world as they expect to; and the simulated virtual world has to react as expected.

As capabilities increase, the range of objects and experiences they can mimic will widen. Valve’s Knuckles controller may let us move beyond hammers and pistols to things like tossing a Frisbee, juggling balls, or grasping a stylus. We’re still years from being able to mimic plucking guitar strings, or tickling a puppy. Still, each step of progress should open an exponentially larger set of possibilities.

Powering interactivity

The flipside of increased input sensing capability is the impact to the computer simulation. More sensors generate more data to process, but that’s only the beginning. As the level of interactivity within the world increases, so will the physics required to drive it realistically.

One peeve of mine is that even in some high-quality, big budget game titles, a character runs up to an item, and it suddenly, magically pops into their hand. It turns out the work involved to make a character’s hand reach for and wrap fingers around objects, is computationally expensive. These details matter because in VR, these gaps are immediately apparent – especially when the hand picking up objects is the user’s!

So VR will drive the need for more physics. Also, the physics will grow in sophistication. The closer users get to objects, the more naturally they can interact with them, and the more they’ll be attuned to expected behavior.

Creating the compute performance required to power all this content, interactivity and physics will present a challenge. But some of that challenge will be offset by computers getting more powerful over time. However, application developers will still have to develop ways to handle scaling across a performance range.

Larger challenges will come as developers grapple with more complex, open-ended simulations. This problem isn’t new. Game developers in particular, evolved through the 2D-to-3D transition, and from small teams to huge blockbuster productions — all while managing more complex and open-ended environments. Technical and design challenges abound, but I’m confident developers will address and overcome them as they have in the past. The platforms most open to innovation will be the places most fertile to solutions emerging.

Toward a rich interactive future

Karen McClure, speaking of interaction design history, said: “Think of it how we learned to speak a foreign language, the language of machines … how over time we taught the machines to speak our language.”

The evolution of interactivity in VR is a work in progress along these lines. We are expanding the vocabulary and grammar with which machines can listen to and speak to us. As that language becomes richer, it’ll become possible to simulate anything, even things we consider impossible. I think that’ll be kind of fun.