With a Wave of the Hand, Improvising on Kinect

In his work as a graduate student at the School of Visual Arts in New York, he created one from thin air.

He hacked a Microsoft Kinect game machine so the device’s cameras follow his body and turn its movements into notes. He can pluck notes from the air as if he were playing a harp.

“Traditional instruments are really, really complex.” he said. “Not only do you need to know how to produce a note, but know how to produce them rapidly in sequence. To be a master musician, you’ve got to be able to pull off all of the micromovements necessary for it to be an expressive performance.”

From the day it was released last fall, the Kinect captured the imagination of experimenters, starting with programmers, roboticists and tinkerers. Research scientists have long used pairs of cameras and specialized algorithms, but Microsoft’s machine handled all the hard work. The Kinect was the first relatively low-cost way to capture three-dimensional information about the world.

Because the Kinect is inexpensive and easy to work with, the experimentation grew more mainstream, and people began sharing their software and reporting on their exploits using sites like kinecthacks.net.

Microsoft developed the device as an attachment for the Xbox game console and built in enough power to track the movement of people and objects in front of it in three dimensions. The Kinect, which costs $150, uses a camera with an infrared light that is not visible to the human eye.

The various objects and people in a room reflect this light and the camera analyzes the pattern from this invisible light. From this it reconstructs a three-dimensional view of the room and tracks any people in front of it.

Thousands of people have used Kinect to create musical instruments, artworks or other projects. Many of their efforts can be found on sites like kinecthacks.net, kinecthacks.com or hackaday.com.

Such sites often outline how to do it yourself. People have created voice-controlled music playback software for televisions, colored on-screen “water trails” that flow from your feet wherever you walk on the floor, animated art inside picture frames as in the Harry Potter movies and a way to turn any surface into a multitouch reading device.

Artists seem to be the largest group drawn to the device, perhaps because there’s not much excitement in waving your arms to manipulate the quarterly budget report.

In one of Alexandra Wolfe’s classes at Carnegie Mellon, she programmed her Kinect to track her movement and add a swirl of colored lights that followed her on screen.

“I was tracking the silhouette because the Kinect has this great user tracking data,” she said. “You can do that with plain computer vision, but it would have taken ages and require so much intense code, but it only took five minutes.”

Anyone who has a new task for the Kinect can experiment with the tools, which are proliferating. Some of the packages require sophisticated programming ability, but others are meant to be simpler.

One of the easier approaches is to download the Flexible Action and Articulated Skeleton Toolkit from the University of Southern California. It converts physical gestures into instructions for PC software.

The toolkit simulates the clicking of a keyboard. This makes it much easier for anyone to use because the keyboard is a common interface.

The toolkit team took a popular game, World of Warcraft, and created a list of gestures that corresponded to keystrokes. Leaning forward, for instance, is converted into a push of the W key that would normally be pressed to walk forward in the game.

Users are already swapping lists of gestures that work well for various software packages.

Using a tool like this makes it fairly easy to reprogram any software that responds to the keyboard. Many photo packages, for instance, have slide shows that advance when the space bar is hit. It’s easy to arrange for any gesture, like a wave of the hand, to be used to skip to the next picture.

The Southern California project site has a forum where people swap lists for common software packages and games. The blog kinecteducation.com also offers a number of suggestions for using the package in classrooms.

All of this excitement has not gone unnoticed at Microsoft, and the company recently released its own software development kit for noncommercial users.

These tools are supporting a wide range of experiments in home automation, art and manufacturing. Gerard Rubio worked with two friends, Raul Nieves and Jordi Bari, in Barcelona to build a three-dimensional copying machine that would scan an item from three sides with three Kinects and then send the model to a three-dimensional printer. A person could stand in the center of the circle and be turned into a sculpture in a few minutes.

Artists are integrating the tool with projectors and sound cards to turn the way we move our bodies into a performance.

Peter Morton, an Australian programmer, worked with a friend to build a simulation of the scene from the movie “Big” in which Tom Hanks and Robert Loggia played “Heart and Soul” and “Chopsticks” on a piano keyboard floor mat with sensors. The Kinect version doesn’t require any floor sensors because it watches the feet.

The team used software from OpenKinect.org and wrote its own software in the Python language to extract the position of the feet and turn it into music.

Mr. Morton said that the OpenKinect software was evolving quickly to be more flexible and supportive, freeing up the programmers to experiment and be creative.

“The sheer scope and variety of demos produced so far could not have been predicted by Microsoft,” he said, and then added with a bit of surprise: “Or anyone else.”

Article source: http://feeds.nytimes.com/click.phdo?i=401085ecfbd0312a5ba62e3f5b73f3ae