Apps and the Apple TV

• July 7, 2011
• 19 notes

The speculation surrounding Apple’s fall announcements has been focused on new versions of the iPhone and iPad, but we’ll undoubtedly see additional “Apple TV will get apps” speculation as the event nears. Developers, the tech press, and even tech savvy users have been fascinated by the possibility of running iOS apps on the Apple TV. And that fascination reached a fever pitch last fall when Apple launched the iOS based Apple TV 2.

This fall Apple may or may not announce a beefed up Apple TV with the ability to download and run apps directly, but even if they do it wont be quite as ground breaking as many assume. The ground has already been broken with AirPlay and AirPlay mirroring in iOS 5.

There remains a fundamental misunderstanding of the difference between mouse-like pointing devices and touch screen interaction. I’ve written about this before, but I’ll do so again—direct manipulation of objects on a touch screen device is a fundamental change in human-computer interaction and is undoubtedly the future of most, if not all, consumer computing devices. The age of the mouse is ending, but the implications are still unclear to most.

When the Magic Trackpad launched, quite a few posts were written about the death of the mouse. The problem with most of those posts is that they conflated the Magic Trackpad with touch screens. The mouse as an object that physically moves around your desk is dead, but fundamentally the Magic Trackpad is just a better mouse-like pointing device. The Magic Trackpad is not the future, it’s just one of the last great pointing devices.

I think the best way to illustrate what I’m trying to say is to tell the story of my 2 year old son, Luke, trying out AirPlay mirroring for the first time…

The other day I was watching the Tour de France on my TV using AirPlay mirroring of NBC’s TDF app running on my iPad 2. When Luke saw the iPad sitting there seemingly unused, he asked if he could play games on it. I did my best to explain to him that the iPad was busy sending video to our TV and was completely blow away by his response. In essence, he asked if he could play his games on the TV. He’s just a few months past his 2nd birthday, but he instantly grokked that the iPad was able to send video to the TV. Why not, he’s growing up surrounded by magic.

What’s interesting, though, is what happened next. When handed the iPad, he looked down at it and launched this week’s favorite app, The Monster at the End of This Book. He looked up at the screen and was excited to see Grover on TV. Then he looked down at the book and flipped the page. Then he looked up and was again excited to see Grover on TV. Then he looked down and turned the page. After just 60 seconds the thrill was gone and he was mostly just playing with the iPad, only intermittently looking up to confirm that Grover was still on TV.

After a few minutes he exited the app and looked up to see the icons of all his favorite apps on the TV. He immediately set down the iPad, walked up to the TV, and tried launching an app by touching the TV screen. My wife and I instinctually told him not to touch the TV, but he looked back at us quite puzzled. The thing is, Luke has never used a mouse-like pointing device. Other than using the TV remote to turn the TV on and off, or turning a light switch on and off, he’s never used one object to remotely manipulate another.

I can’t overstate what a fundamental shift this is. In the entire history of computers we have used a keyboard, keypad, mouse, mouse-like pointers and other similar devices to manipulate objects on a remote screen. [The Palm Pilot and other devices that used a stylus for input did foreshadow the iPhone, but I’d still lump a stylus with mouse-like pointing devices, though I’ll save that argument for another time.] The iPhone changed all that.

As my son so clearly demonstrated, a remote screen is much less interesting when you can directly manipulate objects on a touch screen. The two most obvious conclusions to draw from that statement are: 1. All screens should be touch screens 2. Remote screens will go away because direct manipulation is more compelling. But I think there’s a third, more likely conclusion: TVs and other remote screens will be limited-use extensions of our mobile touch screen devices.

In thinking about how TVs and other remote screens will be used in the future, I think we need to consider how the user interacts with objects on the remote screen and what use-cases are truly more compelling with a remote screen. As my son demonstrated, interacting with the iPad while looking back and forth from the iPad to the TV is onerous and seeing Grover on the TV was fun, but not ultimately compelling.

The current Apple TV user interface isn’t much different from the mouse-like pointing paradigm we’ve grown accustomed to since the introduction of the Macintosh in 1984. The only difference is that you navigate a selected state among objects instead of navigating a pointer over those objects. We can only visually focus on one thing at a time, so when two devices are involved one of them has to be operated without looking, or the user has to look back and forth between the two objects. Someone who doesn’t know how to touch-type is constantly looking down at the keyboard then back at the screen. The mouse doesn’t require us to look at it, but it is a learned skill. Most of us don’t remember, but using a mouse for the first time is disorienting and takes time to master.

Even break-through technologies like Microsoft’s Kinect require remnants of the pointing device paradigm. There is a very physical disconnect between what the user is doing in front of the TV and what is happening on-screen. That disconnect must be somehow mitigated. In most situations, Microsoft has chosen to use ghosted body parts to help connect the user to the on-screen action. The body then becomes just another pointing device with a pointer visible on-screen.

In discussing the iPad, Steve Jobs famously said “If you see a stylus, they blew it.” Similarly, if you see a pointer on-screen, you’re still using a mouse-like device, even if it’s your body that has become the mouse. If the iPad is just a big trackpad that moves a pointer on a remote screen, we’re in for a very boring future. But that’s where things get quite interesting, the touch screen is the iPad’s primary input method in normal use, but the accelerometer, gyroscope, compass, gps, proximity sensor, light sensor, camera, and even the microphone provide options for input that don’t require looking down. With Airplay maturing this fall in iOS 5 and Microsoft officially allowing developers to tinker with Kinect, we’re just now starting to explore the ways in which a remote screen can be used in compelling ways apart from the human-computer interaction paradigms of the past.