A New World of Gaming: Creating Mixed Reality
25 Jun 2011
Our Favorite toys growing up existed entirely in either the real or virtual world. Chess, Monopoly, and even LEGO’s are all made up of real world physical objects that we can touch. Video games, on the other hand, live in the virtual world - think Halo, Portal (my favorite), Call of Duty Black Ops or even Pong. With the invention of Wii and the Microsoft Kinect, we are starting to see some real world elements come into play in the virtual world, based off the player’s gestures.
Sphero is an entirely new concept - one in which game play lives in both the real world and the virtual world at the same time. In 1994, Paul Milgram and Fumio Kishino defined mixed reality as "...anywhere between the extrema of the virtuality continuum."
The graph below simplifies the MR concept. Basically each extreme is what we commonly experience everyday. Using objects in the real world is one extreme and all our computer, smartphone and computer gaming addictions are at the other extreme. Both extremes are outside of the mixed reality experience.
So what’s in the middle and is it really that important?
Let’s look at Wii and Kinect. Both live at the right end of the MR continuum, but are clearly integrating real world interactions into their game play. With Kinect your physical actions in the real world are connected to your virtual character on the screen; if you jump he jumps.
At the left end of the MR continuum is the real world with a virtual world overlay. For example, my lovely and petite wife has trouble seeing what is behind her when backing up - luckily in her new car she has a backup camera. When she backs up not only does she see what’s behind her, she has guidelines that tell her which direction to travel to avoid objects. It even places red blocks on things that seem like expensive collision points - don’t backup into the red blocks or it will cost you.
What this means to you:
Building Mixed Reality experiences is technically difficult and gets even harder as you move closer to the center of the MR spectrum. Why is it so hard? The challenge falls into two main areas - computational complexity and user acceptance.
Essentially, computational complexity means that computers need to do a ton of processing in order to understand the real world. The real world is a messy place. Detecting edges, people, objects and movement in a variety of lighting conditions, with all kinds of things that can get in the way, requires fast processors and lots of hair-pulling math. How can you detect a couch if the lights are off or if there is no standard shape for a couch? How can you tell the difference between a ball and a round leg of a table? Not only do you have to understand the real world in computer terms - you have to understand it in real time so it can be used by an application. Lucky for us, in the last few years, consumer grade computers and smartphones have evolved to the point where they can do enough computations to figure this all out. But we still need more power because with today’s technology it is just barely good enough.
The next challenge is designing applications that are meaningful for us – the user. From our perspective Kinect is very simple - it’s precision is very crude - but technically it is unique. The games and apps have to be designed around the technology and the player has to find that compelling. It would be really cool if the Kinect could detect if I was holding up one finger or two - but it can’t - it doesn’t have that fidelity (yet, maybe soon). So the games have to be designed at a level that work with the technology. It would be great if my wife’s car could back up all by itself without hitting objects or people - but the technology isn’t quite there yet - so we have yellow lines and red blocks.
So what are we trying to do with Sphero?
We’re trying to move closer to the center of the MR continuum with our game play.
Sphero is a real world object with a ton of sensors. Sensors produce data that is easy for a computer to understand. A smartphone can easily process that information and get quite a bit of data on what Sphero is doing. We can then incorporate that information into the game.
Smartphones are also great at virtual world games - plus they are portable and have a camera. The camera is one important element for Sphero’s MR experiences. When you look at Sphero through the camera you are creating a new view of the world. The smartphone can place virtual objects in that view while also identifying where Sphero, a real world object, is in relationship to those objects. This is different than just detecting what is behind a car or if someone is jumping up and down because we can affect the behavior of the real world object, Sphero. We are placing a virtual obstacle in front of a real world ball and seeing how it responds.
As an example, we can put up a virtual wall in front of Sphero and he will not be able to drive through it. In the real world there is no wall, but in the new world view of Mixed Reality (and on the screen of the smartphone), Sphero conforms to the virtual and real world rules. Like Kinect, we have to design game play that works with the capabilities of the technology - but even this basic capability opens up a whole new world of possibilities.
Sphero will have many games and apps that fall along the MR spectrum - some will be very simple, like Sphero following you while you walk. Others will blend the world into a new experience altogether. But even in our simplest app Sphero lives in the MR continuum - control in the virtual world, play in the real (which is opposite of Kinect and Wii).
It can all be very academic to think about the MR continuum. The bottom line is these types of games and applications are changing the way we interact with technology. By blending the two worlds we are opening up a new era of computing - one that will require more power, speed and creativity.