What is creativity, after all? “Creativity is piercing the mundane to find the marvellous,” says Bill Moyers. That is exactly what Pranav Mistry did to create Sixth Sense, an interface between the real world and the digital world, based on some of the everyday human gestures. Mistry saw more in the ubiquitous mouse than just a device to guide the cursor on the computer screen. He opened up two old fashioned mice took out the rollers that guide the balls and made a $2 motion sensing device that can track the gestures made by the hand in the physical world. The result was the computer knew that a photo was to be clicked when he shaped his four fingers like a frame of a picture. When he held a pen and traced a pattern in space, the same pattern got traced on the computer screen.
Next, Mistry mounted a projector on his head using a bike helmet. With the help of this projector, he projected the digital information onto the real world. His aim was to take pixels of primary colours from his phone and colour real objects by projecting the pixels on them. Next, he put a small camera on his body that acted as the digital eye. This helped him interact with the digital world. The camera tracks the hand gestures, which are replicated and projected on any surface, so that one can choose any physical surface to draw digital pictures.
Thus was born the Sixth Sense, “a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with the information.” The Sixth Sense device consists of a camera, a projector and a mirror, hanging from the neck like a pendant. The projector and the camera are connected to the computing device in the user’s pocket.
Wearing the Sixth Sense, one can go sightseeing and simply frame beautiful scenes with thumb and forefingers of both the hands. The Sixth Sense device automatically takes pictures of the framed scenery. Then one can project these pictures on to any wall, crop the pictures, zoom in or out just by finger movement and post the pictures on Facebook. If no surface is available, one can simply use one’s palm as the surface for the projections. Projecting a calculator on one’s palm, one can do a quick calculation or projecting a phone dial, one can connect to one’s friends.
The Sixth Sense camera not only understands one’s hand movements, it also understands the objects held in one’s hands. If one goes to a library or a bookshop and opens a book, the camera will read off the name of the book and project its reviews on the surface of the book. Holding a newspaper in front of the camera, one can see live news projected on the surface of the newspaper.
In order to give sound to the visuals, Pranav Mistry took out the microphone of a webcam and attached it to the paper with the help of a clip. The simple piece of paper came alive with movies and video games. After coming home, he pinched the information from the paper and put it on his desktop. Likewise, information from a book can be pinched out and put on the paper duplicating as a computer screen. Then one can modify that information and take a print out.
The details of Pranav Mistry’s Sixth Sense can be witnessed on his TED Talk. Mistry ends his talk saying, “Integrating information to our everyday objects will not only help us to overcome the digital divide, but also help us, in some way, to stay human, to be more connected to the physical world. And it will actually help us not turning into machines sitting in front of another machine.”
If there is anything uniquely Indian about Pranav Mistry’s way of thinking, it is the ‘jugaadi system’ that is ingrained in the DNA of all Indians. We see this jugaadi system at work with plumbers, car mechanics, electricians, watch repairers and many a subaltern tinkerer who is not educated in the conventional sense, but has an instinctive understanding of how things work. It is this ‘jugaadi’ instinct that prompted Pranav Mistry to open up the mouse and the webcam and put the components to a different use. He crossed the boundaries of convention. He created miracles with the mundane.
The current Sixth Sense prototype system costs approximately $350 to build. Instructions on how to make your own prototype will be available on www.pranavmistry.com/projects/sixthsense/.
Pranav Mistry is currently heading the Think Tank Team and is the Director of Research of Samsung, where he’s busy developing the latest in Samsung’s arsenal of cutting edge technologies.
Compiled by Subha Das Mollick