The study of facial expressions - a brief history
Reading facial expressions has become a science since the days of Charles Darwin. Doing it well is also an art, as I hope you will see in using this website. Doing it accurately is hard mental work, but let’s not forget that it is also a pleasure.
In the summer of 1868 Charles Darwin liked to show guests a collection of photographs of facial expressions and ask them to identify the emotion. The guests were fascinated by this parlour game, even if those by now familiar prints by the neurologist Guillaume Duchenne made them feel rather queasy. The torturous images were created using electrodes on the facial muscles of an old man who suffered from a condition which numbed his facial nerves. The photograph of the old man’s facial muscles being electrically stimulated into giving the appearance of a smile has become a classic, still published widely. A genuine smile involving crinkling around the eyes (rather than the fake 'Pan Am smile') is rather confusingly known as a Duchenne smile.
For Darwin, the new technology of photography was an exciting opportunity to examine facial expressions systematically. Victorian natural history enthusiasts were driven by the urge to collect and classify (despite the fact that the photographic process in the early years was painstakingly slow and the results uneven). The new technology of photography allowed Charles Darwin to add the examination of human facial expressions to his skill in creating taxonomies of stuffed or dried animal species.
Darwin’s parlour game was typical of the Victorian era, with its fascination with the new technology of photography and the human image, together with the passion for collecting and organising. But it was also a more serious scientific endeavour. Darwin’s contribution, among many others, was to define emotional expressions as a special type of behaviour, not simply ‘the passions’ as they were commonly known. He saw facial expressions as signals which our ancestors would have developed to help ensure their survival.
While photography enabled Charles Darwin to study facial expressions in a scientific way in the 1860s, interest in the subject can be traced back to Greek philosophy and the earliest writings on physiology. In the second century the Greek physician Galen proposed that the study of expression should be a medical subject. For ancient physicians and philosophers, Victorian neurologists, and naturalists alike, physiognomy was never simply a biological study. The fascination with what faces are saying to us is part of being human. Reading facial expressions is an essential human skill and an ability which has helped in the development of the human species. Being able to transmit and receive emotional states would have been highly advantageous to our ancestors. The mirroring act of displaying fear and reading it on another individual’s face would help hunters to respond to danger and increase their chances of survival.
The origins of human facial expressions
Seventy-five thousand years ago, according to the neuroscientist Vilayanur S. Ramachandran, human beings made an enormous developmental and cultural leap through the development of a sophisticated mirror neuron system. Mirror neurons enabled human beings to place themselves in relation to others; mimic what they did, intrinsically learn from others’ actions, and thereby spread practical knowledge such as tool use, creating fire, building shelters, and rearing animals.
The mirror neurons enable us to feel empathy with others. Reading facial expressions is an essential component. When we move into the sphere of facial expressions we are entering into the most social of human activities. Recent research has shown that to successfully read facial expressions we must work our own facial muscles microscopically to mimic the expressions we see in others. This explains the surprising finding that people who have had Botox treatment to freeze facial muscles find it more difficult to interpret the facial expressions of others. It also explains why children who have been given dummies or soothers as infants can find it more difficult to read facial expressions when they are older, since their ability to mimic the faces of others has been inhibited by the soother in the mouth. The sheer energy spent in making microscopic facial muscle changes explains why people on my training courses in facial coding are exhausted at the end of the day, in a way they would not be if they were simply doing a mental activity. Reading facial expressions is a real physical activity, as well as a deeply social and empathetic one, requiring emotional work.
The study of other social primates is an invaluable aid in understanding our ancestors and our development. Dutch primatologist and ethologist Frans de Waal emphasises our social nature giving rise to human development as we fulfilled the first need of security. As primates we are interdependent for our security. It is our membership of the group which provides security from predators; it is our communication which binds the group together.
Like the apes at play we humans excel at bodily synchrony; falling into the same stride when walking together, or unconsciously mirroring each other across a table. We revel n exercising this skill - in football crowds, for example, the Mexican wave has become an art form.
We will never know definitively how facial expressions evolved, but it’s a fair assumption that they began as the primary means for babies to communicate with their mother (with 3D scanning technology we can now see the baby in the womb appearing to practice a smile). Emotional expressions started out as something evolutionary biologists call a 'cue' - they revealed information about an inner state or behaviour. Then, over many generations, they evolved into deliberately emitted signals for expressly conveying information within groups, in combination with the grunts and sounds which ultimately developed into language.
The specific facial expressions associated with each particular emotion appear to have evolved for a reason. The squinched-up nose and mouth of disgust make it harder for you to inhale anything poisonous drifting on the breeze. The out-thrust chest of pride increases both testosterone production and lung capacity so you are ready to take on anyone.
Darwin proposed that emotion expressions evolved to serve two classes of function: (a) preparing the organism to respond adaptively to environmentally recurrent stimuli and (b) communicating critical social information. Subsequent researchers further developed this account, arguing that internal physiological regulation was likely the original adaptive function of emotion expressions. So, for example, our eyes open wider when we experience pleasure and narrow when we want to block out threatening or confusing objects from view. Later these evolved to signal to our companions when we are happy and relaxed, or puzzled - thereby serving important communicative functions.
Although it is hard to discern a function for some expressions, perhaps the original purpose has long been obscured as the expressions have changed and evolved. Another possibility is that some emotional expressions only ever served as a signalling function. Pride and shame, which are particularly social emotions, are likely candidates. The expressions resemble the dominance and submission postures of other primates and other mammals - dogs for example. We can easily see how some facial expressions and postures are signals inherited from our distant ancestors living in small groups with a well-defined status hierarchy, or pecking order.
It is commonplace to credit language as the key to the development of the human brain, but it is worth reflecting on the role facial expressions have played. They add subtlety and complexity to verbal communication. Sometimes as we watch actors in a drama it is the facial expressions which convey the strongest message (think of Bob Hoskins in the closing scene of The Long Good Friday).
A biological response and a social tool
Scientists generally agree that expressions evolved to elicit behaviours from others. So, a smile may encourage people to approach, while a scowl may warn them to stay clear, and a pout may elicit words of sympathy and reassurance. We see the same signals in primates, as de Waal has noted in chimpanzee colonies.
Thinking of facial expressions as tools for influencing social interactions provides an opportunity to begin predicting when certain facial expressions will occur and will allow more precise theories about social interactions. Studies find that expressions occur most often during pivotal points in social interactions - during greetings, social crises or in times of appeasement, for example.
This implies that expressions of emotion become more intense, not only when we feel the emotion more strongly, but when we are most keen to persuade the person we are addressing.
Although we can see how facial expressions serve as a communications tool, they are not confined to social situations. We show our emotions when we are alone, as well as when in the company of others.
Not all emotional expressions have obvious functions, however. The original biological purpose of happy smiles, angry scowls or sad frowns have so far eluded psychologists. We can speculate that expressions became more exaggerated and distinctive over time in human development to make it easier to communicate non-verbally.
Paul Ekman - creating an academic discipline
The technology of photography enabled Darwin to study facial expressions systematically. It was the use of video which enabled Paul Ekman to conduct his study in Papua New Guinea to test the universality of facial expressions.
Working through a series of studies with his long-time friend Wallace V. Friesen, Ekman found a high level of agreement between diverse Western and Eastern cultures when asking people to select emotional labels to fit a number of different facial expressions.
Ekman demonstrated that the findings extended to preliterate tribesmen in Papua New Guinea, whose members could not have learned the meaning of expressions from exposure to media depictions of emotion. In the late 1960s, a team of psychologists led by Paul Ekman of the University of California, travelled to Brazil, Japan, Borneo and New Guinea and showed people photographs of six stereotypical emotional expressions: happiness, fear, anger, surprise, disgust/contempt and sadness. They also did the experiment at home in the US.
Ekman’s team found that everybody they tested, regardless of culture, recognised the same six basic emotions - even people in Borneo and New Guinea who had no prior contact with the outside world. “For most of these people, I was the first outsider they’d seen,” Ekman recalls.
Since then, dozens of studies have seen similar effects. In addition to the basic six (or seven, with contempt and disgust sometimes treated separately), the model has been extended to include pride, indicated by a tall posture and puffed-up chest, and shame, with a downturned head and bent posture. This all supports the view that emotional expressions are hard-wired into the human brain.
In a study of footage of seeing and blind athletes at the 2004 Olympic and Paralympic Games, psychologist David Matsumoto at San Francisco State University analysed the facial expressions of sighted and blind judo contestants, including athletes who were blind from birth. He found that all three groups produced the same faces when they won a bout. The expressions included the so-called Duchenne smiles - big, beaming smiles involving the eyes as well as the mouth - which are considered to be authentic expressions of happiness. “The evidence for the universality of facial expressions of emotion is overwhelming,” says Matsumoto.
Are there differences between cultures?
The authors are not asserting that emotions are fixed and unvarying and unaffected by culture or social context. Ekman and Friesen later demonstrated that certain emotions were exhibited with very specific ‘display rules’: culture-specific prescriptions about who can show which emotions to whom and when. We can think of the most obvious examples in the appropriate display of emotions at a funeral, for example; and the contrast in the wake when the mourners have a licence to show mirth, which would be shocking during the ceremony. But display rules permeate society at a more subtle level - some emotional displays are acceptable in some workplaces depending on the particular culture, some not. Most of us make small unconscious adjustments to our expressions of emotion, depending on the context we are in and the mood of the people around us.
When we think of how emotions are displayed in different cultures we can easily think that the emotional lives of people in those cultures are different from our own. We can assume, for example, that the Japanese are ‘buttoned up’ or Italians impulsive and passionate. I experienced this prejudice quite recently when looking at footage of interviews I had conducted in China. In commissioning the study I wanted to demonstrate that facial coding can be done just as effectively on different ethnic groups (for example, Chinese) by a Western analyst. However, when the video footage of the first interviews came through I was dismayed to find that the respondents, affluent females in their thirties, responded to the questions in a guarded way, showing little emotion and averting their gaze from the interview. I found myself looking at the stereotypical female from the Far East - modest, inscrutable, restrained.
Fortunately, however, I had arranged to review the study with the female owner of the research agency. The early interviews had been conducted by men; we decided that she should conduct the remaining interviews. The response to a female interviewer was a stark contrast to how they responded to the male interviewers. The women were relaxed, open, ready to joke, offering very frank opinions (and when the translations came through I discovered they were also quite comfortable using salty language).
Rather than help my analysis, this sensitivity to racial differences exposes the risk of being influenced by cultural stereotypes. This example is a good illustration of how cultural differences lie not in the use of facial muscles, but in the display rules which can conceal the universality of facial expressions.
Different people express themselves differently
As well as the different norms influencing facial expression between cultures, there are also important differences between individuals. For example, what about the physical ability to make expressions? Interestingly, there is quite a large physiological difference between individuals when it comes to facial muscles. At least potentially, some people have a greater expressive ability than others.
The first study on facial muscles done by examining human cadavers was conducted in 2008. The study, by Bridget Waller at Portsmouth University in collaboration with anatomists in the US, found that there are 19 facial muscles, each one of which may not be present in all people. For example only two-thirds of the population have the risorius muscle that creates an expression of terror. In some individuals the anatomists found only 60 per cent of the available facial muscles. The implication of this is that some individuals will physically be able to express their feelings more than others.
However, like other parts of the body, the facial muscles can adapt to form expressions even though the exact musculature is different. And we can, with sufficient effort, teach ourselves to control parts of the face, even when we assume it is a genetic ability - raising one eyebrow, for example. We simply don’t know how much of the variation between individuals in expressiveness is due to physical limitations. However, even in what appears to be a straightforward physiological difference, it is likely that social and psychological factors are at play. A child who is not encouraged by parents to express emotions, will be unlikely to develop as broad a range of expressions as a child who is encouraged to express their feelings, verbally and non-verbally.
Like many aspects of human communication we also see the result of learned behaviour according to gender. Although males, conventionally, are regarded as less expressive than females, this may be partly due to social norms. For example, we expect women to smile more and in fact, it has been established that they do smile more. By contrast, men are expected to show more anger, but in reality they don’t.
Certainly in the case of primates, as Bridget Waller has discovered, apes who live within large social groups have a more elaborate ‘communication repertoire’ including more complex facial expressions. We can see a link between cultural norms about expressiveness on the one hand and practice in using (and even developing) facial muscles on the other.
The varying ability to read facial expressions
Research into children on the autism spectrum provides the largest collection of academic research data on the ability to read emotions. Autistic children, even when gazing directly on to the face, fail to recognise visual cues and don’t process some of the critical information. Typically, autistic children miss the nuances of non-verbal communication - the body language and facial expressions - and instead focus on minor, less socially relevant details.
The efforts to help those with autism has been the motivation behind developing a facial expressions training guide for those on the autism spectrum led by Simon Baron-Cohen at the Autism Research Centre. A computerised facial expression system was originally developed by MIT professor Rosalind W. Picard and research scientist Rana el Kaliouby to help children with autism understand facial expressions. (Recently we have also developed a tool for children on the autism spectrum, an website called Emotions Reader).
Within the general population, my experience tells me that around 40% of people can read facial expressions well and around 20% have a poor ability to detect emotions through reading faces. Tests of facial reading ability tend to be fairly crude, focusing on Ekman’s six basic emotions and using still photographs. Academic research has shown that movement is required to accurately identify an emotion; a few frames of video showing the expression being formed and leaving the face makes a substantial difference (as we see in the video footage in this website).
Also, in the real world, the facial expressions we commonly encounter are weak, elusive or blended, resulting in a signal that often requires interpretative work. Some individuals don’t have the ability or the inclination to do this work. In my experience this interpretative work nevertheless reveals a great deal of agreement between those viewing the emotions.
My work training facial expression analysts has shown me that individuals agree to a large degree on what emotion they are seeing and precisely when. This is after looking carefully at video footage and not, as is the weakness of many academic studies, looking at static images.
The proviso to this is that those who do not have the facility are screened out of the process. I do not have evidence of ‘naturals’ as Ekman has posited. However, there is no doubt that some people are poor at reading facial expressions. This is consistent with Baron-Cohen’s idea of male and female brain differences and the autistic spectrum. The ability to read facial expressions is generally considered to be more of a female attribute, but like most cited differences between ‘male’ and ‘female’ brains, there is considerable variation.
The importance of context clues
When we analyse facial expressions we have an important advantage if we know what preceded the expression - were they talking about a sad event, or reflecting on a difficult question?
We read expressions partly by putting ourselves in the shoes of others, which triggers our emotional intelligence. Facial coding involves careful examination of the myriad configurations of the face, but it is more than a simple visual assessment of a collection of facial muscles, the angle and movement of the head.
In consumer research context is relatively easy since most expressions follow a stimulus, such as a verbal proposition or brand name, so we expect certain emotions. When we describe an odd smell we can expect disgust rather than, say, anger. Similarly, in our social world, when we talk to a friend about a bad experience we have an idea of the emotions which might be forthcoming.
Experiments conducted by academics have found that people’s recognition of an emotion depends heavily on context. In the real world, faces are rarely seen in isolation. Posture, tone of voice, other faces in the wider context are also available clues which influence how expressions are interpreted. For instance, a scowl - usually associated with anger - can be disgust if the person is holding a dirty object, or fear if it’s paired with a description of danger. The disgusted face can even be pride if attached to a body with arms raised in triumph and a puffed-out chest.
Microexpressions - feeling and thinking
One of Ekman’s greatest achievements has been to draw our attention to microexpressions; the fleeting expressions of emotion, some of which can last a fraction of a second. Although Haggard and Isaacs were first to coin the term for fleeting emotional expressions, Ekman developed the concept and made the term well-known.
In this website there are examples of fleeting expressions of emotion (and thinking), demonstrating how quickly they can come and go. But the main value of capturing them is that they reveal the activity of the unconscious brain, or System 1 type thinking. If we wish to understand a person’s real emotions and thinking, capturing their facial expressions at the moment they are presented with a stimulus (for example a press ad, a design or a product feature) provides the best means available to see the working of their unconscious brain.
The gaze tells us about thinking
We use our eyes to see when someone is thinking or not thinking. For example, we can tell when our companion is thinking about what we are saying, or their mind is engaged on something else. An attentive partner will ask: ‘what are you thinking about?’. A child will notice if their parent is not giving them their full attention. A good teacher will notice when the mind of the student goes somewhere else. We notice when someone’s thoughts go elsewhere. This is not only in obvious cases when their gaze goes over our shoulder in a crowded room. We can also sense when someone is gazing at our face, but their mind is wandering. A gaze into the middle distance indicates the brain is working on thoughts unrelated to the object of the gaze, whether that is a person or an advertisement. Showing a focused gaze on the object, indicating activity of the brain, is what we term unconscious engagement.
The power of the unconscious brain
There has been a plethora of books discussing the now established fact that much of our activity is guided by the large and powerful unconscious brain. This unconscious working brain is named System 1 by Kahneman, bottom-up thinking by Goleman and automaticity by Bargh and others, but it brings us to the same conclusion. Much of what we do in everyday life, the multitude of small and large decisions we make, we do automatically, without making a conscious decision. As Kahneman says, we cling on to the idea of having conscious control, but that is illusionary. The conscious brain is rather like a supporting character who believes himself to be the hero.
Modern neuroscience has shown us that there are many things going on under the surface of our conscious thought which enable us to function in the world. Brain scanning technology has revealed just how quickly the unconscious brain makes decisions. In a ground-breaking study by Soon and his colleagues it was demonstrated how the unconscious brain can reach a decision up to 7 seconds before the person is consciously aware of having made a decision.
We like to believe that we go through life consciously making decisions on the basis of free will. But in reality ‘free will’ takes on a different meaning when we spend most of the time driven by automatic processes. A trip to the shop requires hundreds of small decisions - what we look at as we walk down the street, who we recognise, when to cross the road. When we are in the shop, our unconscious brain influences what words, symbols and colours we are drawn to on the shelves.
In a 2013 BBC science documentary, a number of neuroscientists were asked to imagine that a sheet of paper represented the mind and were asked mark out the proportion involved in conscious decision-making. Every one of the scientists drew a small area, no more than an eighth of the page.
It is both fascinating and disconcerting to think that we make decisions all the time without even being aware of it. Fascinating because it leads us to see ourselves in a fundamentally different way, not as a rational conscious brain in charge of a responsive body. Our mindful state is part of a complex whole with multiple feedback mechanisms. It no longer makes sense to draw a simple distinction between mind and body. Nor does it make sense to distinguish between the emotional and the rational. The Freudian model of Super Ego, Ego and Id has a poetic plausibility and a comforting conclusion: we may be subject to the base instincts of the Id, but we are capable of controlling those
impulses. The new neuroscience tells us that a lot of what makes us function in the world - including thousands of small decisions - is taken care of automatically.
The idea of automaticity is disconcerting because ultimately we have to accept that our unconscious brain is only partially understood by us. We are not strangers to ourselves; but we’re not on the closest of terms either. David Eagleman likens our conscious brain to a tiny stowaway on a transatlantic steamship, taking credit for the journey without acknowledging the massive engineering underfoot.
In my experience in consumer research, when people tell us about their intentions we really should treat their verbal data with a fair degree of scepticism. It has been said that when we are waiting for considered responses, we are waiting for the White House spokesman to speak, not the President.
Facial expressions - a key to the unconscious brain
The study of facial expressions, like most scientific subjects, has been through peaks and troughs from Darwin, to Ekman. Now we see a renewed interest through the field of neuroscience. Scientists recognise that facial expressions are an indispensable way to gain access to the workings of the unconscious brain.
This has had major consequences in the field of consumer research. We now accept that what someone might tell market researchers about their buying intentions using their conscious brain is unreliable. Now that practitioners know how important System 1 thinking is in understanding the behaviour of consumers, there is a race in the market research and advertising industries to capture the workings of the unconscious brain. A number of methods have been tried, ranging from EEG headsets to monitor electrical activity on the scalp (and thereby brain activity), to bio vests which monitor heart rate, blood pressure and galvanic skin response (and thereby physiological and psychological arousal).
The most powerful and reliable of these is facial coding. We can see the results of brain activity within a fraction of a second as signals go directly to the muscles of the face and body. Involuntary facial expressions come directly from the unconscious brain and are part of what neurologists call the extrapyramidal system, a neural network responsible for involuntary reflexes and movement. These involuntary microexpressions can begin and end in 200 milliseconds.
The discovery of mirror neurons
Facial expressions are a prime means of connecting the mind with the body. They are also an important way of connecting people.
We don’t exist in our own independent moods; we are all interconnected, both bodily and emotionally. The discovery of mirror neurons has been hailed as of the same importance to psychology as the discovery of DNA has been for biology.
Mirror neurons, which fire both when you do something and when you watch someone else do the same thing, allow us to mimic others. This tendency to map the actions of others onto our own bodies is thought to play a role in our ability to understand others’ intentions and emotions, boosting empathy.
Those who are good at interpreting facial expressions have been found to have a greater number of active mirror neurons.
It has long been thought that our facial expressions influence, not only how others around us feel, but how we ourselves feel.
As early as the 1890s, psychologist William James promoted his facial feedback hypothesis. James believed there was a causal link between making facial expressions and the subsequent emotional state. A number of studies a century later, in the 1980s and ‘90s, support this proposition. Now we have firm evidence that our own facial expressions do influence how we feel. Smiling can make us feel better, while frowning can make us feel worse. M. Axel Wollmer and colleagues at the University of Basel in Switzerland, reported that paralysing the glabellar frown region - the area between the eyebrows - with Botox, significantly improved symptoms in depressed individuals.
The study included 30 participants, each of whom had symptoms of major depression and were unresponsive to traditional anti-depressive medications. Fifteen individuals were assigned to receive a dose (five injections) of Botox, while the other fifteen received placebo (saline) injections.
Six weeks after this single dose, individuals were assessed for depressive symptoms. The researchers found that those who received Botox injections experienced a 47% decrease in depressive symptoms, compared to just 9% in the placebo group. This improvement remained over the 16-week study period.
Smiling itself produces a weak feeling of happiness. In one study, participants were instructed to hold a pencil in their mouths, either between their lips or between their teeth. The latter group, who were able to smile, rated cartoons funnier than did the former, who could not smile because of the pencil between their lips.
The mind-body connection
Research also provides evidence of a causal link between motion and emotion, by showing that bodily movements influence the recollection of emotional memories, as well as the speed with which they are recalled.
The eminent neuroscientist Antonio Damasio proposes that the feelings produced by the physical states of the body come to guide behaviour and decision-making. When something bad happens the brain leverages the entire body to register the feeling (heart rate, contraction of the gut, weakness of the muscles and so on). When the event is next pondered, the brain essentially runs a simulation, reliving the physical feelings of the event.
This kind of embodied cognition is a relatively new concept, but there is no doubt that we involuntarily enter the minds and bodies of others, allowing us to recreate what we have seen others do.
A number of experiments have involved pasting electrodes onto subjects' faces and registering the tiniest of facial movements. Within a time too quick for conscious response, scientists flashed images of facial expressions onto the screen. Asked what they had seen after this slide show, the subjects had no conscious awareness that they had seen frowning and smiling faces, but they had still frowned in response to angry faces and turned up the corners of their mouths in response to happy faces. Not only that, but the subjects shown happy faces reported feeling better after the experiment than those shown frowning faces.
Moving our facial muscles, making expressions, like other physical actions, has an effect on the brain, not just the other way round. The relationship between the brain and movements of the body is not one-way. Rather surprisingly, by changing the direction of our eyes, for example, we can improve our problem-solving ability. Scientists have found that by occasionally guiding the eye movements of participants with a tracking task unrelated to the problem, they were able to substantially improve their chances of solving the problem to the point where those groups outperformed every control group.
The same researchers have also shown how getting people to swing their arms in a certain way helped them to solve a physical problem involving connecting strings hanging from the ceiling in a room. The theory is that by making the subjects swing their arms in a particular way, a part of the brain that deals with swinging motions is activated and unconsciously leads them to think about that type of motion when trying to solve the problem.
People tend to think that their mind lives in their brain, dealing in conceptual abstractions, very much disconnected from the body. This emerging research demonstrates how the body is a part of the mind in a powerful way. The way you think is affected by your body and, in fact, we can use our bodies to help us think.
The study of the mind-body connection is only just beginning to develop, but we now know that there is a complex relationship between interpreting and conveying facial expressions and how we think and feel.