CECS Spotlight: Look into my eyes

How computers are being used to recognise anxiety

It's a familiar scenario for many computer gamers: You're playing your favourite shoot-em-up game, working you way through a level you've already played and it all seems so... familiar. Adversaries appear from the same old places. You have to negotiate your way around the thoroughly recognisable obstacles. In short, it's all a bit boring.

But in the not-too-distant future, according to one ANU researcher, computer games could come equipped with programs that sense the mood of the gamer. So if the game is becoming boring, it could ramp up the excitement, or if the pace of the action is bringing the gamer to the brink of seizure, it could slow it down to something more manageable.

That's one of the applications that Gordon McIntyre, a PhD student from the Research School of Information Services and Engineering, sees for this work into technology that can detect a person's emotional state by analysing their speech and facial expressions.

McIntyre has 20 years experience of working with computers and now uses all that knowledge in the area of 'affective computing'. This term was made famous by Rosalind Picard at Massachusetts Institute of Technology, meaning "computing that relates to, arises from, or deliberately influences emotion or other affective phenomena."

But where previous attempts to undertake similar tasks have involved people being wired to the computer, McIntyre's work will see their moods monitored without that level of attachment.

He says there are two major inputs; people's speech and their faces. "From the speech signals we look at the rhythm and the pitch," he said. "It's trying to measure the energy levels and comparing that to what could be seen as a typical energy level with the rhythm and pitch that someone would emit in a certain emotion. "With facial expressions we're taking an average face and looking at how various expressions might deviate from that and then trying to sense what the person's emotional state is." With the range of emotions that a person can feel and the huge deviations in how that can be represented in a person's speech of facial expressions, McIntyre has sharpened his focus to look for just one state - anxiety.

This one simple-sounding emotional state actually comprises many different things, as he explains. "Anxiety disorder is an umbrella term that covers quite a few different conditions. There are the social and phobia type anxieties, general anxiety disorder - where people are anxious about something for more than six months - then there are anxieties like Obsessive Compulsive Disorder and panic attacks. There are a lot of terms included under the term 'anxiety'.

"Anxiety is basically like fear, but different because the threat is less tangible. With fear the threat is usually imminent, whereas anxiety tends to be less tangible and your response or reaction to the threat is usually out of proportion to the threat itself." But the complexity of undertaking the project isn't just confined to many faces of anxiety. McIntyre's use of motion video to capture a person's emotional state has made the task a great deal trickier than previous attempts, which looked at static images.

"Motion video is a much harder task," he said.

"There's really a few different stages: one is the facial expressions of still images, then moving beyond that to try and look at a moving face - which is more difficult given the short durations and bursts a person might give in their emotional appearance. Finally there is the talking face, which is the ultimate. I'm trying to do it in steps."

McIntyre has had to be resourceful in finding the work of others and building it into his own model. This has included picking up work from other researchers at ANU who looked at how to map points with topical features on the face, software others have developed around facial expressions, getting help with some complicated mathematics and reading. Lots and lots of reading.

"When I did my degree 20 years ago I did it by correspondence," McIntyre says.

"If I wanted a book I'd have to get a microfiche, look up the book, send away a request and they'd send it on to me. These days it's the exact opposite - I get on the Internet and there's so much information available and so much software. It's a huge challenge to understand it all. You can start reading all the references to something and several hours later you've realised it's not really what you wanted in the first place."

But researching dead-ends haven't been the only thing that might bring a furrowed brow. With the huge demands on computers needed to pull off something like facial expression recognition, McIntyre's research is something that can only be achieved by modern, super-fast PCs. "Computer performance is ever-increasing, but it has only been in the last few years that the performance of the everyday workstation computer has become adequate to run some of this software.

"Then there is the challenge getting the different pieces of software to integrate - that's been one of the bigger challenges. There is a large amount of software available from different universities with different levels of documentation, plus commercial software," he said. According to McIntyre, however, the challenges are worth it for the potential outcomes. One of the ways he sees his software being applied is helping with the care of patients with dementia.

"One way it could work is sensing the emotional state of people with dementia when they can't tell you themselves. "It could look at the effectiveness of medicine in those cases by giving you an idea of their emotional state before and after - so you use a computer to measure a person's emotional state and then check it again after medication."

McIntyre says his work is just a small example of how computers will play a more significant role in peoples' lives in the future.

"Computers of the future will be much more pervasive," he says.

"Bill Gates talks of robots for things like minding the kids and taking the washing out and so on. I think beyond the gimmickry there's certainly an opportunity.

"I went to a conference last year where one of the world's foremost computer visionaries talked about some of the projects he's looking at. Where he sees the use of computers is not to substitute human contact, it's to fill the gap between what a person wants to do and what they're capability is. You don't want computers to take over what you do, you want them to assist you."

But if, in the future, a computer can recognise your emotional state, it leaves the possibility open that the technology could be used in a variety of ways - including areas that could be considered obtrusive or even infringing on someone's civil rights.

It's a possibility that McIntyre acknowledges, but thinks that is less likely to happen.

"Fortunately in Australia people are quite cognisant about the need for privacy and one would hope that those privacy principles would be extended and equally applicable to this.

"But it's not just about privacy; it's also that you don't want to take over human interaction from people. People can relate to computers, but they're still no substitute for human interaction.

"You do hear about applications that people use for marketing. One of the cornier examples is billboards that are used to advertise perfume which judge viewer's facial expression to determine if they like what they're seeing, but that isn't the focus of my project."

Those concerns are still some time away, so for now it's on with developing the complicated software necessary to recognise anxiety - and dealing with how people react when McIntyre tells them what he's researching.

"Usually people start talking about their phobias - such as their fear of snakes, spider or aeroplanes," he laughs.