SIXTH SENSE TECHNOLOGY
Although
miniaturized versions of computers help us to connect to the digital
world even while we are travelling there aren’t any device as of now
which gives a direct link between the digital world and our physical
interaction with the real world. Usually the information’s are stored
traditionally on a paper or a digital storage device. Sixth sense
technology helps to bridge this gap between tangible and non-tangible
world. Sixth Sense device is basically a wearable gestural
interface that connects the physical world around us with digital
information and lets us use natural hand gestures to interact with this
information .The sixth sense technology was developed by Pranav Mistry, a
PhD student in the Fluid Interfaces Group at the MIT Media Lab. The
sixth sense technology has a Web 4.0 view of human and machine
interactions. Sixth Sense integrates digital information into the
physical world and its objects, making the entire world your computer.
It can turn any surface into a touch-screen for computing, controlled by
simple hand gestures. It is not a technology which is aimed at changing
human habits but causing computers and other machines to adapt to human
needs. It also supports multi user and multi touch provisions. Sixth
Sense device is a mini-projector coupled with a camera and a cell
phone—which acts as the computer and your connection to the Cloud, all
the information stored on the web. The current prototype costs around
$350. The Sixth Sense prototype is used to implement several
applications that have shown the usefulness, viability and flexibility
of the system.
2. DEFINITION
Sixth Sense' is a wearable gestural interface that augments the
physical world around us with digital information and lets us use
natural hand gestures to interact with that information. By using a
camera and a tiny projector mounted in a pendant like wearable device,
'Sixth Sense' sees what you see and visually augments any surfaces or
objects we are interacting with. It projects information onto surfaces,
walls, and physical objects around us, and lets us interact with the
projected information through natural hand gestures, arm movements, or
our interaction with the object itself. 'Sixth Sense' attempts to free
information from its confines by seamlessly integrating it with reality,
and thus making the entire world your computer. All of us are aware of
the five basic senses – seeing, feeling, smelling, tasting and hearing.
But there is also another sense called the sixth sense.
It is basically a connection to something greater than what their
physical senses are able to perceive. To a layman, it would be something
supernatural. Some might just consider it to be a superstition or
something psychological. But the invention of sixth sense technology has
completely shocked the world. Although it is not widely known as of now
but the time is not far when this technology will change our perception
of the world.
The Sixth Sense prototype is comprised of a pocket projector, a mirror
and a camera. The hardware components are coupled in a pendant-like
mobile wearable device. Both the projector and the camera are connected
to the mobile computing device in the user’s pocket. The device projects
visual information, enabling surfaces, walls and physical objects
around the wearer to be used as interfaces; while the camera recognizes
and tracks the user's hand gestures and physical objects using
computer-vision based techniques. The software program processes the
video stream data captured by the camera and tracks the locations of the
colored markers at the tip of the user’s fingers using simple
computer-vision techniques. The movements and arrangements of these
fiducials are interpreted into gestures that act as interaction
instructions for the projected application interfaces. The maximum
number of tracked fingers is only constrained by the number of unique
fiducials, thus Sixth Sense also supports multi-touch and multi-user
interaction.
The Sixth Sense prototype
implements several applications that demonstrate the usefulness,
viability and flexibility of the system. The map application lets the
user navigate a map displayed on a nearby surface using hand gestures,
similar to gestures supported by multi-touch based systems, letting the
user zoom in, zoom out or pan using intuitive hand movements. The
drawing application lets the user draw on any surface by tracking the
fingertip movements of the user’s index finger. Sixth Sense also
recognizes user’s freehand gestures (postures). For example, it
implements a gestural camera that takes photos of the scene the user is
looking at by detecting the ‘framing’ gesture. The user can stop by any
surface or wall and flick through the photos he/she has taken. Sixth
Sense also lets the user draw icons or symbols in the air using the
movement of the index finger and recognizes those symbols as interaction
instructions. For example, drawing a magnifying glass symbol takes the
user to the map application or drawing an ‘@’ symbol lets the user check
his mail. The Sixth Sense system also augments physical objects the
user is interacting with by projecting more information about these
objects projected on them. For example, a newspaper can show live video
news or dynamic information can be provided on a regular piece of paper.
The gesture of drawing a circle on the user’s wrist projects an analog
watch. The current prototype system costs approximately $350 to build."
The device sees what we see but it
lets out information that we want to know while viewing the object. It
can project information on any surface, be it a wall, table or any other
object and uses hand / arm movements to help us interact with the
projected information. The device brings us closer to reality and
assists us in making right decisions by providing the relevant
information, thereby, making the entire world a computer.
The world has shrunk. Distances have dissolved. Communication lines and
interaction with countless systems have been rendered feasible. However
this technological overhaul has been peripheral and not so much related
to the human body; researchers and innovators have constantly grappled
with the issue of bridging the gaps which limit the human-environment
contact. This device, tentatively name as the Sixth Sense,
is a wearable machine that assists unexplored interactions between the
real and the virtual sphere of data. It consists of certain commonly
available components, which are intrinsic to its functioning. These
include a camera, a portable battery-powered projection system coupled
with a mirror and a cell phone. All these components communicate to the
cell phone, which acts as the communication and computation device. The
entire hardware apparatus is encompassed in a pendant-shaped mobile
wearable device. Basically the camera recognizes individuals, images,
pictures, gestures one makes with their hands and the projector assists
in projecting any information on whatever type of surface is present in
front of the person. The usage of the mirror is significant as the
projector dangles pointing downwards from the neck. To bring out
variations on a much higher plane, in the demo video which was
broadcasted to showcase the prototype to the world, Mistry uses colored
caps on his fingers so that it becomes simpler for the software to
differentiate between the fingers, demanding various applications. The
software program analyses the video data caught by the camera and also
tracks down the locations of the colored markers by utilizing single
computer vision techniques. One can have any number of hand gestures and
movements as long as they are all reasonably identified and
differentiated for the system to interpret it, preferably through unique
and varied. This is possible only because the ‘Sixth Sense’ device
supports multi-touch and multi-user interaction. There was once a clear
divide between the virtual world and the real world, but that line is
getting blurrier every day.
3. GESTURES
The software recognizes three kinds of gestures:
Ø Multi-Touch Gestures:
Like the ones we see in the iphone-where we touch
the screen and make the map move by pinching and dragging.
Ø Freehand Gestures:
Like when you take a picture or Namaste gesture to start the projection on the wall.
Ø ICONIC Gestures:
Drawing a icon in the air. Like, whenever we draw a star, shows us the
weather details. When we draw a magnifying glass, shows us the map.
4. COMPONENTS
The devices which are used in sixth sense technology ra :
1. Camera
2. Coloured Marker
3. Mobile component
4. Projector
5. Mirror
Ø Camera:
It
captures the image of the object in view and tracks the user’s hand
gesture. The camera recognizes individuals, images, pictures, gestures
that user makes with his hand. The camera then sends this data to a
smart phone for processing. Basically the camera forms a digital eye
which connects to the world of digital information.
Ø Coloured Marker:
There
are color markers placed at the tip of users finger. Marking the user’s
fingers with red, yellow green and blue coloured tape helps the webcam
to recognize the hand gestures. The movements and arrangement of these
markers are interpreted into gestures that act as a interaction
instruction for the projected application interfaces.
Ø Mobile Component:
The
SixthSense device consists of a web enabled smart phone which process
the data send by the camera. The smart phone searches the web and
interprets the hand gestures with help of the colored markers placed at
the finger tips.
Ø Projector:
The
information that is interpreted through the smart phone can be
projected into any surface. The projector projects the visual
information enabling surfaces and physical objects to be used as
interfaces. The projector itself consists of a battery which have3 hours
of battery life.
A
tiny LED projector displays the data sent from the smart phone on any
surface in view- object, wall or person. The downward facing projector
projects the image on to a mirror.
Ø Mirror:
The
usage of a mirror is important as the projector dangles pointing
downwards from the neck. The mirror reflects the image on to a desire
surface. Thus finally the digital image is freed from its confines and
placed in the physical world.
5. HOW IT WORKS
The Sixth Sense prototype is comprised of a pocket projector, a mirror and a camera. The hardware components are coupled in a pendant like mobile wearable device. Both the projector and the camera are connected to
the mobile computing device in the user’s pocket. The projector
projects visual information enabling surfaces, walls and physical
objects around us to be used as interfaces; while the camera recognizes
and tracks user's hand gestures and physical objects using
computer-vision based techniques. The software program processes
the video stream data captured by the camera and tracks the locations
of the colored markers at the tip of the user’s fingers using simple
computer-vision techniques. The movements and arrangements of these fiducials are interpreted into gestures that act as interaction instructions for the projected application interfaces.
The maximum number of tracked fingers is only constrained by the number
of unique fiducials, thus Sixth Sense also supports multi-touch and
multi-user interaction.
Both the projector and the camera are connected to the
mobile computing device in the user’s pocket. The projector projects
visual information enabling surfaces, walls and physical objects around
us to be used as interfaces; while the camera recognizes and tracks
user’s hand gestures and physical objects using computer-vision based
techniques. The software program processes the video stream data
captured by the camera and tracks the locations of the colored markers
at the tip of the user’s fingers using simple computer-vision
techniques. It also supports multi touch and multi user interaction.
The technology in itself is nothing more than the combination of some stunning technologies.
The technology is mainly based on hand gesture recognition, image
capturing, processing, and manipulation, etc. The software of the
technology uses the video stream, which is captured by the camera, and
also tracks the location of the tips of the fingers to recognize the
gestures. This process is done using some techniques of computer vision.
However, instead of requiring you to be in front of a big screen like
Tom Cruise, Sixth Sense can do its magic—and a lot more—everywhere. The
camera recognizes objects around you instantly, with the micro-projector
overlaying the information on any surface, including the object itself
or your hand. Then, you can access or manipulate the information using
your fingers. The key here is that Sixth Sense recognizes the objects
around you, displaying information automatically and letting you access
it in any way you want, in the simplest way possible. Clearly, this has
the potential of becoming the ultimate "transparent" user interface for
accessing information about everything around us. If they can get rid of
the colored finger caps and it ever goes beyond the initial development
phase, that is. But as it is now, it may change the way we interact
with the real world and truly give everyone complete awareness of the
environment around us.
The Sixth Sense technology works as follows:
1. It captures the image of the object in view and track the user’s hand gesture.
2. There
are colour markers placed at the tip of users finger. Marking the
user’s fingers with red, yellow green and blue coloured tape helps the
webcam to recognize the hand gestures. The movements and arrangement of
these markers are interpreted into gestures that act as a interaction
instruction for the projected application interfaces.
3. The smart phone searches the web and interprets the hand gestures with help of the coloured markers placed at the finger tips
4. The information that is interpreted through the smart phone can be projected into any surface.
5. The mirror reflects the image on to a desire surface.
6. TECHNOLOGIES RELATED TO SIXTH SENSE TECHNOLOGY
Ø Augmented Reality:
The augmented Reality is a visualization technology that allows the
user to experience the virtual experience added over real world in real
time. With the help of advanced AR technology the information about the
surrounding real world of the user becomes interactive and digitally
usable. Artificial information about the environment and the objects in
it can be stored and retrieved as an information layer on top of the
real world view. When we compare the spectrum between virtual reality,
which creates immersive, computer-generated environments, and the real
world, augmented reality is closer to the real world. Augmented reality
adds graphics, sounds, haptic feedback and smell to the natural world as
it exists. Both video games and cell phones are driving the development
of augmented reality. The augmented systems will also superimpose
graphics for every perspective available and try adjust to every
movement of the user’s head and eyes. The three basic components of an
augmented reality system are the head-mounted display, tracking system
and mobile computer for the hardware. The main goal of this new
technology is to merge these three components into a highly portable
unit much like a combination of a high tech Walkman and an ordinary pair
or eyeglasses. The head-mounted display used in augmented reality
systems will enable the user to view superimposed graphics and text
created by the system. Another component of an augmented reality system
is its tracking and orientation system. This system pinpoints the user’s
location in reference to his surroundings and additionally tracks the
user’s eye and head movements. Augmented reality systems will need
highly mobile computers. As of now many computers aren’t there to
satisfy to provide this option.
Ø Gesture Recognition:
It is a technology which is aimed at interpreting human gestures with
the help of mathematical algorithms. Gesture recognition technique
basically focuses on the emotion recognition from the face and hand
gesture recognition. Gender recognition technique enables humans to
interact with computers in a more direct way without using any external
interfacing devices. It can provide a much better alternative to text
user interfaces and graphical user interface which requires the need of a
keyboard or mouse to interact with the computer. Interfaces which
solely depends on the gestures requires precise hand pose tracking. In
the early versions of gesture recognition process special type of hand
gloves which provide information about hand position orientation and
flux of the fingers. In the SixthSense devices coloured bands are used
for this purpose. Once hand pose has been captured the gestures can be
recognised using different technique’s. Neural network approaches or
statistical templates are the commonly used techniques used for the
recognition purposes. This technique have an high accuracy usually
showing accuracy of more than 95%. Time dependent neural network will
also be used for real time recognition of the gestures.
Ø Computer Vision:
Computer vision is the technology in which machines are able to
interpret/extract necessary information from an image. Computer vision
technology includes various fields like image processing, image analysis
and machine vision. It includes certain aspect of artificial
intelligence techniques like pattern recognition. The machines which
implement computer vision techniques require image sensors which detect
electromagnetic radiation which are usually in the form of ultraviolet
rays or light rays. The computer vision find itself applicable in varies
field of interest. One such field is bio medical image processing. It’s
also used in autonomous vehicle like SUV’s. The computer vision
technique basically includes four processes.
1. Recognition:
One of the main task of computer vision technique is to determine
whether the particular object contain the useful data or not.
2. Motion
Analysis: Motion analysis includes several tasks related to estimation
of motion where an image sequence is processed continuously to detect
the velocity at each point of the image or in the 3D scene.
3. Scene
Reconstruction: Computer vision technique employs several methods to
recreate a 3D image from the available images of a scene.
4. Image
Restoration: The main of aim of this step is to remove noise from an
given image. The simplest method involves using simple filters like low
pass or median filters. In order to get better quality images more
complex methods like Inpainting are used.
Ø Radio Frequency Identification:
Radio frequency identification systems transmit the identity of
an object wirelessly, using radio magnetic waves. The main purpose of a
radio frequency identification system is to enable the transfer of a
data via a portable device. The portable device is commonly known as
tag. The data send by the tag is received and processed by a reader
according to the needs of the application. The data send by the tag
contains various information’s identification or location of the
information, or specifics about the product that has been tagged, for
example price, colour, date of purchase, etc. This technology gained
importance because of its ability to track moving object. A typical
radio frequency tag consists of a microchip attached to a radio antenna
which is mounted on a substrate. To retrieve the data from the tag a
reader is needed. A typical radio frequency reader consists of two
antennas that emit radio waves and at the same are capable of accepting
the signals from the tag. The reader then passes the information that it
has received to a computer device in digital form. The computer then
interprets this digital data and processes it. Radio frequency
identification techniques are widely used in the fields like asset
tracking, supply chain management, manufacturing, payment systems etc.
One of the major advantages of devices using radio frequency technology
over other similar devices is that RFID devices need not be positioned
precisely relative to the scanner. But till then there are certain areas
of concern for this technology. Some problem related to this technology
is tag collision and reader collision. Usually the reader collision
occurs when the signals from two or more readers overlap, while tag
collision occurs when many tags are present in a small area.
7. APPLICATION
The
sixth sense technology finds a lot of application in the modern world.
The sixth sense devices bridge the gap by bringing the digital world
into the real world and in that process allowing the users to interact
with the information without the help of any machine interfaces.
Prototypes of the sixth sense device have demonstrated viability,
usefulness and flexibility of this new technology. According to the
words of its developers the extend of use of this new device is only
limited by the imagination of human beings. Some practical applications
of the sixth sense technology is given below
Ø Viewing Map:
With the help of a map application the user can call upon any map of
his/her choice and navigate through them by projecting the map on to any
surface. By using the thumb and index fingers movements the user can
zoom in, zoom out or pan the selected map.
Ø Taking Pictures:
Another application of sixth sense devices is the implementation of a
gestural camera. This camera takes the photo of the location user is
looking at by detecting the framing gesture. After taking the desired
number of photos we can project them onto any surfaces and then use
gestures to sort through those photos and organize and resize them.
Ø Drawing Application:
The drawing application allows the user you to draw on any surface by
tracking the fingertip movements of the user’s index finger. The
pictures that are drawn by the user can be stored and replaced on any
other surface. The user can also shuffle through various pictures and
drawing by using the hand gesture movements
Ø Making Calls:
We can make calls with help of sixth sense device .The sixth sense
device is used to project the keypad into your palm and the using that
virtual keypad we can make calls to anyone.
Ø Interacting with physical objects:
The SixthSense system also helps to interact with physical objects we
use in a better way. It augments physical objects by projecting more
information about these objects projected on them. For example, a
gesture of drawing a circle on the user’s wrist projects an analog watch
on the user’s hand. Similarly a newspaper can show live video news or
dynamic information can be provided on a regular piece of paper
Ø Getting Information:
Sixth sense devices can be used for
getting various information relating to our everyday life by getting in
contact with objects
· Product Information:
Sixth sense technology uses marker technology or image recognition
techniques to recognize the objects we pick in our hand and then provide
information relating to that product.
· Book Information:
By holding and shuffling through the book pages, the sixth sense
provides Amazon ratings on that book, other reviews and other relevant
things related to the book.
· Flight Updates:
With the help of the sixth sense technology it is no longer required to
log into any sites for checking the status of the flights. The system
will recognize your boarding pass and let you know whether the flight is
on time or not.
· Information About People:
With help of face recognition techniques the sixth sense devices are
able to provide information about the people when we meet them. The
sensor detects the face and checks the data base for the relevant
information. The system will then project the relevant information about
a person like what they do, where they work,
8. ADVANTAGES
Ø Portable:
One of the main advantages of the sixth sense devices is its small size
and portability. It can be easily carried around without any
difficulty. The prototype of the sixth sense is designed in a such a way
that it gives more importance to the portability factor. All the
devices are light in weight and the smart phone can easily fit into the
user’s pocket.
Ø Support multi touch and multi user interaction:
Multi touch and multi user interaction is another added feature of the
sixth sense devices. Multi sensing technique allows the user to interact
with system with more than one than one finger at a time.. Sixth sense
devices also in-cooperates multi user functionality. This is typically
useful for large interaction scenarios such as interactive table tops
and walls.
Ø Cost effective:
The cost incurred for the construction of the sixth sense proto type is
quiet low. It was made from parts collected together from common
devices. And a typical sixth sense device cost up to $300. The sixth
sense devices have not been made in large scale for commercial purpose.
Once that happens it’s almost certain that the device will cost much
lower than the current price.
Ø Connectedness between real world and digital world:
Forming a connection between the real world and the digital world was
the main aim of the sixth sense technology.
Ø Data access directly from the machines in real time:
With help of a sixth sense device the user can easily access data from
any machine at real time speed. The user doesn’t require any
machine-human interface to access the data. The data access through
recognition of hand gestures is much easier and user friendlier compared
to the text user interface or graphical user interface which requires
keyboard or mouse.
Ø Mind map the idea anywhere:
With the advent of the sixth sense
device, requirement of a platform or a screen to analyze and interpret
the data has become obsolete. We can project the information into any
surface and can work and manage the data as per our convenience..
Ø Open source software:
The software that is used to interpret and analysis the data collected
by the device is made open source. This enables other developers to
contribute to the development of the system
9. CONCLUSION
Sixth
Sense recognizes the objects around us, displaying information
automatically and letting us to access it in any way we need. This
prototype implements several applications that demonstrate the
usefulness, viability and flexibility of the system, allowing us to
interact with this information via natural hand gestures. This became
the ultimate “transparent” user interface for accessing information
about everything around us.
vd :)