Sphero Augmented Reality

08 Nov 2012

Download Sharky The Beaver for iPhone and iPod or iPad.

Gaming is always evolving, and right now it’s headed in an exciting direction – towards augmented reality (AR). This technology refers to overlaying a live view of the real world with virtual content. It’s an interactive breakthrough made possible by our mobile devices, and it’s changing the tech landscape in a big way. You have probably heard of several examples of augmented reality in the past year, such as Google’s mysterious Project Glass. For gaming applications in particular, the goal is to render 3D content into the live scene so that it appears to be realistic and authentic. That’s where Sphero rolls in.

Augmented reality gameplay with Sphero is simple and seamless for two reasons – Sphero is a robot, and Sphero is round. This allows us to employ a robotic fiducial, which provides never-before-seen freedom in characterizing the visual environment (since by definition a robot is a reprogrammable machine). With no auxiliary fiducial markers, Sphero can be effectively found and tracked at a distance, at any angle, and when partially obscured. A ball is the same from every angle, after all.

Sphero AR also has the ability to impact the real world, whereas traditional augmented reality is limited to the screen. Not only can we make Sphero interact with virtual objects (collect coins, run over monsters, etc.), but we can also make Sphero respond to these objects (bumping into invisible walls, hitting fictional oil slicks). Just wait till you’re holding Sphero in your hand, looking at an augmented reality character through the screen of your device. It’s pretty awesome.

Mixing robotics with augmented reality does present some interesting challenges. Not only are we dealing with 3D graphics, but we must also combine data from the sensors, motors, and the observed motion of Sphero to make complex inferences about the location of the ball and the player. At the end of the day, putting together all the sensor data from Sphero and your mobile device, analyzing the video feed, setting up the lighting, camera, and action takes some heavy-duty math.  Let's break it down.

1.  Data starts out in Sphero's accelerometer and gyroscope (get the low-down on Sphero’s inner robot here) and is piped into a state of the art, sensor-fusion algorithm. This generates Sphero's sense of orientation. The combination of theses sensors and software behaves very much like Sphero's "inner ear".  It allows Sphero to know which way is up (literally) and which way it's facing. You may get the spins on a merry-go-round, but Sphero can handle hundreds of RPM without getting dizzy.

2.  Next, the camera feed from your mobile device is captured and combined with the device's own sense of orientation. This gives us a rough understanding of the contents of the picture.  Are we looking at the ground, or at the sky?

3.  Here comes more math. Custom vision algorithms tear the image apart, identifying Sphero, identifying the floor, analyzing the color of the lights, the ground, and any other data that might impact feedback.

4.  Data from the motors is then combined with Sphero's sense of orientation to generate a sense of position. Just like you can navigate parts of your house with your eyes closed, Sphero can combine its sensor data to figure out where it is.  And you might be surprised how good this sense is. You can tell Sphero to drive a twisting path through a large room, and it still knows how to drive back to within inches of its starting location.  I challenge you to try that blindfolded!

5.  All of the sensory information is combined using a lot of geometry, statistics, and duct-tape to figure out the real-world location of Sphero, the height of the player, the position of virtual objects, and the direction and color of lights.

In the end, the fluid, responsive augmented reality experience with Sphero boils down to the accuracy of about a dozen numbers describing the real and virtual scene. Even with carefully optimized code, it takes hundreds of millions of calculations per second to compute these.  If that seems like a lot of effort for a dozen numbers, just imagine what's going on in the visual cortex of your own brain. A not-so-simple process, resulting in seamless and intuitive gameplay with endless possibility.

What can you expect as Sphero ventures into the new frontier of augmented reality? More apps, more fun, and a lot more entertainment. Get ready to play like never before. Stay posted for announcements on Sphero augmented reality and peeks behind the scenes on our Facebook page and Twitter at @GoSphero.  You can also join our newsletter when you visit our new homepage at GoSphero.com.

Subscribe to our newsletter to keep updated on news, promotions and all things Sphero.
Ready to upgrade your play?

Visit our online store for Ollie or Sphero as well as all of our great accessories.