While Johnny Chung Leeís name might not ring a bell, his work with the Wiiís Remote has been viewed millions of times on YouTube. Leeís innovative projects use the input device in some mind-bogglingly clever ways, from using it to track usersí fingers, to creating inexpensive multipoint interactive whiteboards (which is much, much cooler than it sounds), and creating a head-tracking system for VR displays. If youíre not familiar with these videos, you need to watch them immediately before reading on.
We spoke with Lee about his projectsówhich started off as a procrastination tool from his PhD studiesóand his thoughts on how his ideas could affect gaming in general. With the attention the Wii has received for its intuitive and interesting game controls, itís great to see someone push hard and inspire others to see that the system lives up to its potential.
Game Informer: Can you talk a little about your background and what led to your work with the Wii?
Johnny Lee: Iím a PhD graduate student at Carnegie Mellon University studying human-computer interaction. Iím actually just finishing up my PhD. Iíve been doing a lot of work and am in a community that focuses on new ways to interact with computers, either by using innovative software or hardware technology. The Wii Remote stuff actually started off as sort of a procrastination project, because itís not my primary thesis topic. I was basically playing with it to do something on the side. The Wii Remote has a lot of really cool devices in it, and itís an easy platform to experiment with. So these are just projects that I thought would be fun to explore personally, and it turns out that a lot of other people find them interesting as well.
GI: Have you found that the Wii stuff has dovetailed into your PhD thesis work at all?
Lee: A little bit. Toward the end of my thesis work I was looking for high-performing tracking technology. It turns out that the camera in the Wii Remote is pretty good and is relatively easy to work with. So, for some of the demos late in my thesis work, I talked about how it could be done with something like the Wii Remote. My work uses a lot of infrared-light-sensing technology, so itís also relevant to the Wii Remoteónot quite in the same way, but itís relevant.
GI: Did you gravitate toward the Wii because of the system itself, or was it simply a good way to get all of the components in a relatively inexpensive package?
Lee: Probably more of the latter. I do play video games and I like video games, but I was much more excited about the capabilities of the controller than I was the Wii game console itself. Itís amazingly impressive that Nintendo just allowed the ability to connect it to a computer over Bluetooth, which I think has been great for both the hobby community as well as the research community. Itís a pretty amazing piece of engineering, containing a high-speed infrared camera, accelerometer, plus it has Bluetooth communication, a vibration motor, speaker, buttonsóand it does this all running off two AA batteries for a long time. If you had talked about this a few months before it was released, I probably would have said it could not have been done, given my exposure to research. But they did it, and I was massively impressed.
GI: Have you worked with other game tech in the past?
Lee: A little bit. Thereís sort of the consumer game technology and then thereís the experimental game technology that you sometimes see at conferences like SIGGRAPH [Special Interest Group on GRAPHics and Interactive Techniques] and UIST [User Interface Software and Technology]óthese are sort of the big either industry or academic research communities, and sometimes they have prototypes of things you could do in the future but arenít necessarily mass market yet. Thereís things like eye tracking and 3D trackers and face-recognition systems. All of the high-end ones are pretty difficult to work with or expensive. Things that are at a consumer level are typically normal game controllers, with joysticks and buttons, so there wasnít anything too interesting there to play with. So the Wii Remote was the first consumer product that I thought that was really interesting to play with.
GI: So you never messed around with the NES Power Glove or anything?
Lee: No, not really.
GI: As a video-game player, what do you think of the gaming landscape now? Do you think thereís an acceptable level of innovation going on in the industry?
Lee: I definitely think that, at least among the traditional gaming platformsómaybe not on the Wii, but on the Xbox and the PlayStationóthe way theyíre going about providing a gaming experience is sort of an asymptotic experience, where theyíre doing a very good job at it, but theyíre not really evolving very far beyond that. To me, probably the most interesting stuff thatís going to happen to gaming in the future will be with new ways to control or interact with the system, either physically or using input devices. So actually I think games are one of the most interesting places to explore new computer-interaction techniques. I think the Wii is a particularly good step toward that, but I thing thereís still a long way to go.
I think developers are really excited about trying to explore these new ideas, but some of the skill sets that I think are good at making joypad games are going to take a while to appreciate and flesh out the capabilities of new controllers. Even on the Wii, you can see that some of these games are essentially joypad games that are retrofitted to the Wii controller. It works, but itís not necessarily the best use of the controller.
GI: Have you played around with the Sixaxis controller at all?
Lee: Yes, I played with it. The games that I played that do use the accelerometer in it are cute, and theyíre pretty well done, but I think the physical aspect of the joypad controller as it is has a certain affordance to the way it wants to be held and used. Itís sort of limited to roll and tilt. I think the remote-control form factor of the Wii lends itself better to more freeform movement.
GI: One of the most interesting things in your videos is how you occasionally put the Wii Remote on a stand and actually manipulate the Wiiís sensor bar as an input device, which is something a lot of people wouldnít think to do. Was that a matter of it being an obvious way of doing that for you, or did you have a classic ďeureka!Ē moment?
Lee: I think it was a little bit of both. It took a while to sort of get over the concept of what the Wii Remote is pitched as, which is as a controller. Part of what I do in my work is I take a device and try to break it down into its subcomponents. If you just think about it having a high-resolution camera, thereís a lot of computer vision work thatís been done in the research environment, but it hasnít made it into the consumer domain. The camera that the Wii provides is pretty impressive. Once I thought about just using the camera component and not as a controller, it opened up a whole bunch of ideas that people may have already done but that most consumers havenít been exposed to. Once I decided to do that, there were probably 10-ish ideas that I could easily do that are all pretty cool.
GI: Are we going to see any of those ideas soon?
Lee: Hopefully. Soon might be hard, because Iím getting a lot of phone calls and Iím also in the middle of my job search, which means that Iím going to be doing interviews. Unfortunately, the next thing may not come out for a few months. Weíll see. This has stimulated a lot of interest among some really smart people who are technically minded, so once you make the leap of using it as a camera, then a lot of other ideas start flying out. Essentially, Iíve inspired a lot of really smart people to start working with the Wii Remote, so hopefully weíll see a lot of cool projects by a lot of people.
GI: Have you talked with anyone at Nintendo about your work?
Lee: I havenít officially talked with anyone at Nintendo. To some degree, I think itís almost in their best interest not to officially endorse or condemn what Iím doing, simply because the game industry is very hype-oriented, so if they say anything about it, people will jump all over it and expectations might get set in the wrong direction. I know a bunch of developers who are working on Wii games are interested in exploring it seriously, so we might see some games, hopefully sooner than later.
GI: If you could get Nintendo to make one significant change to its hardware, what would you ask for?
Lee: Thatís a hard question. Thereís nothing that immediately jumps out to me, I think mostly because itís an odd question to askóitís a very weird hypothetical, and I think the industry is already trying to catch up to the capability that it has built into it. Granted, it may not be the most powerful graphically, but from an interaction standpoint the accelerometer is being underused in games. Most people are using it for shake recognition. The pointer, the camera, has the capability to track up to four points, but most of the games are only using two points. Also, most of the games are only using one controller per person, but you could also use two controllers per person and get some more control out of that. So thereís a lot of stuff that theyíre not doing that it could do. If anything, they already have more capabilities than they need at the moment.
GI: Do you think this is an area that you can see yourself exploring for a long period of time, or is it mostly a lark?
Lee: It kind of depends on how much people want me to continue doing stuff like this. My goal is essentially to have the work I do affect peoplesí lives in consumer products and have people play with it. If thereís a lot of interest in further developing these Wii concepts or working on similar concepts, then I might spend more time on it. There may not be a lot more headroom along the products youíve already seen, because theyíre concepts. Itís up to content developers to make experiences around using these techniques. That gets into skill sets that I donít personally have. Iíll probably move into another interactive experience that hopefully is accessible to a lot of people.
GI: These next few questions are related to your head-tracking video. Do you think that kind of tracking would be especially difficult to implement in a game? It seems as though it might be taxing once applied to a 3D world.
Lee: From a pure technical standpoint, itís trivial. My code uses Direct X and there are maybe only 100 lines of code to make that work. Assuming that the game consoles have the capability of doing that type of renderingóbecause itís a slightly different type of rendering to make it look like itís really 3D; you can move the camera around, but thatís not quite what Iím doingóif theyíre willing to and they have the software code to do that and theyíre willing to bundle some hardware, like IR glasses, then itís trivial for them to make a game. Whether or not itís perhaps taxing as a game player to use it for a long time, Iím not entirely sure yet. There seems to be a lot of enthusiasm around making games that get people moving around and getting involved in the experience, so it might be tiring but that might be a good aspect of playing the game.
Itís up to the developer also to determine how to apply this technique. In software, they can apply a scaler, and small movements translate to big movements in a game, so they can choose or leave it up to player preference. So itís essentially head-motion sensitivity that you would set. From a technical standpoint, it would be easy to make a game, but itís up to the developers to use it properly.
GI: Does the head tracking only work in first-person experiences, or would it be effective in a StarFox type game or in God of War-like gameplay?
Lee: I think itís most compelling in a first-person point of view, but itís still very useful for third person or other gamesóeven in a side-scrolling platforming game. What it does is give you motion parallax, which is the technical term, which is essentially the way objects occlude with each other when you move your head. So it gives you a better sense of space, which any game that has space involved in it could probably benefit from a head-tracking system.
GI: Do you think 3-dimensional technology is the next step in entertainment? It seems like itís something that people revisit every so often, but it hasnít been used in any widely successful applications.
Lee: I think a lot of people are really interested in trying to bring realistic 3D experiences into the home, because itís sort of the next step beyond television. I think where itís been sort of a difficult or an uphill battleóor even a failureóin the past is because usually it requires specialized hardware thatís not in everyoneís home already. The nice thing about the Wii is that itís already in 13-million homes, at least, so most of the hardware is there to do this.
Iíve heard from some technical developers that this is re-exciting the 3D-technology community, and even people who are working on 3D shutter glasses are wondering if they can make them for the Wii Remote or the Nintendo Wii console. Thatís entirely speculation at the moment, but essentially itís a chicken and the egg problem. If you can get the hardware in the homes to do this, then you can make content for it. Hopefully, I think itíll come. Itíll be up to how well people can adopt the technology if itís successful or not.
GI: Do you have any advice for people who want to learn how to do this kind of work on their own?
Lee: Well, if you want to do human-computer interaction stuff, like Iíve been doing, you need to know how to program first. If youíre in high school, take computer classes or participate in project activities outside of school, because applying the knowledge you have in a useful manner really helps you see how the world works and how technology works. I also personally have an electrical-engineering background, which is probably boring if you think about it, but it gives you the skill sets to really change everything about a computer. It allows you to not take the hardware or the software for granted. So if the hardwareís not right, you can change the hardware. And you can tell it what to do with software. And then you can make any experience you can imagine with those skills.
Maybe the answer you donít want to hear is going to engineering school and doing really well in engineering, but most importantly itís probably holding onto your imagination so you can execute your ideas.