CECS Spotlight: Melding mind & machine

A visionary PhD student is developing technology that will respond to our moods and mental processes.

You finish the report by 11.45am and rush to the street. It takes no time at all to find a taxi and traffic is light on the seven-block ride to the specialist's office. As you bound up the steps to the waiting room, a minute before noon, you think that things are finally looking up. You tell the receptionist your name, but she stares at you blankly. The doctor is away this week at a family funeral, she says. Didn't you receive the notice in the post? You remember the pile of letters on the hallway table. The cut on your finger begins to throb.

But then your PDA chirps gently. You pull it from your coat pocket. Text appears on its soothing blue screen, suggesting that you need a break and that there's a cafe around the corner that serves your favourite kind of tea. It also tells you that your friend lives nearby, one you haven't seen for a while, and that perhaps what you really need is someone to whom you can tell your troubles. The clever device offers to reschedule your 3 o'clock meeting and puts off reminding you about your outstanding bills until tomorrow.

Stressed workers would no doubt snap up such an intelligent, emotionally attuned device. Unfortunately, it doesn't yet exist outside the realm of dreams. Fortunately, one dreamer wants to make it a reality.

"It's an idea I've had for a little virtual brain in a personal assistant that you carry around with you," says James Sheridan, a PhD student at the Department of Computer Science and Information Technology (CSIT). "Maybe there are sensors melded into your hat, or parts of your clothes, that are able to monitor your brainwaves or heart rate. The device will know that you're quite stressed. It will know from your schedule that you've got an hour free. The device may have a GPS, so it will know where you are and what's around you. You still have control, but the computer's suggestions are going to be much more contextualised to your life."

The impetus to create technology that would be a better fit for people rose out of Sheridan's need to create a research project that would suit his own circumstances. Despite a record of scholarly success, the young student found it difficult to concentrate on certain activities for great lengths of time. After seeking medical advice, he was diagnosed as having Attention Deficit Disorder (ADD).

"During my Honours year I struggled a lot with the write up," Sheridan says. "I found with reading and writing documents that there wasn't a lot of feedback in the process; where you could do something, look at the results and get some feedback on how you've gone and try something different quickly. But I could concentrate on something like programming or video games, where you've got a very short feedback loop and lots of information to keep you on track."

Intent on pursuing further study, Sheridan negotiated a doctorate that spans the realms of computer science and art. His research project is shared between CSIT and the Centre for New Media Arts at ANU. This meant that his assessment could include elements of a performance and an installation, rather that being solely based on a written dissertation. It also meant that he could pursue his singular take on the arts.

"The question of what is art, what makes art, is very different for different people. Traditional art is staring at paintings, which I find very boring. I would only be able to get into certain types of paintings or painters. Traditional music is the same. But other types of art are very engaging. I've always been interested in creating music from a different view point to do with play, to do with interaction - its always excited me."

As well as creating a form of art that could 'tune' itself to human thought, Sheridan was also interested in creating something that would have therapeutic advantages. He says that his own experience with ADD demonstrated that traditional treatments weren't always effective. He tried neurofeedback, which uses electronic sensors and a visual or aural component to demonstrate brain activity as it happens. This form of therapy is intended to foster an awareness of thought processes, helping people with conditions like ADD to create coping strategies. But Sheridan says typical neurofeedback exercises tend to be very long and repetitive - two factors that could deter someone who has trouble with concentration. "I thought that something like neurofeedback could be done in a much more involved way, where it's part of your everyday practice."

To this end, Sheridan is developing what he calls a "mental Zen garden". This will allow the interaction of human and machine via sensory equipment and clever software. The user will be placed in a virtual reality laboratory that can track human vision and movements while creating three-dimensional graphics. Meanwhile, special head gear containing electroencephalography (EEG) sensors will read electrical activity in the user's brain, feeding information about thought processes into the equation. All these readings will be fed into computers, which will represent the data as a visual and sound display. In short, mental processes will be converted into an evolving garden where developing thoughts trigger animated branches and environmental sounds.

"As you find elements of the growing structures that interest you, you'll be able to focus on those and have those come to the front of the display," Sheridan says. "The artwork will be feeding back into your brain, so that what you're thinking about will affect what is being created.

"Your attention will dictate the sound of things and the aesthetics. It will look and sound very different depending on whether you're someone who looks around a lot or someone who can meditate and focus. Different structures will represent different kinds of brain activity. You'll be able to see how the structures relate to one another and to your attention patterns, and then be able to change the ways you think."

Thought processes will be associated with different characteristics of sound such as reverberation, which humans are conditioned to respond to in certain ways. The overall effect will be of something evolving and responsive, making the way that we currently interact with computers seem positively crude. To get there, Sheridan's project has grown to include several other graduate students who are helping to create the software that will track movements and thought processes. Sheridan says his role is to tie all the various bits of data together into a cohesive whole. It's a challenging task, but one he wants to continue even beyond the scope of his PhD.

"I've never really aspired to be an academic. But the longer I spend at it, the better it seems. I have so many ideas that I want to get built. I've seen the PhD as a way of testing out some of those and taking them somewhere."

If his drive continues, Sheridan's mental Zen garden may yet blossom into a new interface between people and machines.