The Impact of Gestures on Social Intelligence

VR and AR rely on interactions from the participant. Being inside of VR/AR experiences requires the user to take action. Users in these open-loop systems have partial control over the experience by moving their eyes, head, hands, or legs. In the PSVR game Headmaster, the player moves his head in order to bounce incoming soccer balls into the net and score points. It’s an easy example of gameplay relying on player movement.

In Headmaster in PSVR, you act like a soccer player and use your head to spike the ball into the net and score points

There are important implications from relying on human movement for gameplay. Here I’ve collected some research on how gestures specifically affect people’s cognitive capabilities. From Science of People:

• You’re born to speak with your hands. Researchers have found that infants who use more hand gestures at 18-months old have greater language abilities later on. Hand gestures speak to great intelligence.

• Hand gestures make people listen to you. Spencer Kelly, associate professor of Psychology and co-director of the Center for Language and Brain at Colgate University found that gestures make people pay attention to the acoustics of speech. Kelly said, ‘Gestures are not merely add-ons to language — they may actually be a fundamental part of it.’

• We can’t help it. Hand gestures come to us naturally. Spencer even found that blind people use hand gestures when speaking with other blind people. After studying native English and Turkish speakers as well as blind and sighted people, the researchers established that people learn gestures from language and grammar NOT from watching others.

• Gesturing helps you access memories. Using hand gestures while you speak not only helps others remember what you say, it also helps you speak more quickly and effectively!

• Nonverbal explanations help you understand more. One study found that forcing children to gesture while they explained how to solve math problems actually helped them learn new problem-solving strategies.

And from Psyblog:

• Gesture for persuasion — The way people’s hands cut through the air while they talk is…more than just a by-product of communication. Maricchiolo et al. (2008) found that hand-gestures help increase the power of a persuasive message when compared to no use of gesture. Most effective are gestures which make what you are saying more understandable. For example, when referring to the past, point behind you.

There are two additional studies on the impact of gestures on thought:

• fMRI brain imaging studies suggest that people tend to look for meaning in gestures when observing others. This means that VR/AR designers should be thoughtful when creating characters that gesture and limit gestures to things that are relevant and understandable to the user.
• Gestures activate the Mirror Neuron System in the brain, which is associated with mimicry and empathy in social relationships. Mirror neurons are activated when the user makes gestures or when observing others do the same

Takeaways for designers:

• Gestures activate cognitive processing at a deeper level than just words alone
• The right gestures can make VR/AR feel more immersive, memorable, or persuasive
• Don’t add gestures in just for fun. People look for meaning inside of gestures so limit their use in a way that helps people grasp information more quickly