Ode to Audio

Every quarter is different, but there's always this very distinctive turning point about four weeks in, when you go from "i'm so excited about this course I'm taking!" to "JUST GET ME OUT OF HERE ALIVE." in the span of like 3 days. I think this might be day 1 of 3, so I thought I'd be nice and blog before I joined the dark side. Also, for the record, professors who give out easy introductory assignments and follow it with a second, massively harder assignment after the drop deadline, are evil.

So, one of the courses that I am really excited about this quarter (!) is Serge's object recognition course. Not that I am all of the sudden interested in computer vision... but there has actually been some ongoing research that tries to apply methods from (visual) object recognition to auditory object recognition. Briefly: object recognition determines if some specific object appears in a certain image; auditory object recognition determines whether some specific auditory "event" (i.e. spoken word, musical melody, sound of a door slam) has occured in a given audio clip, usually with a noisy environment.

For me, auditory anything was pretty much a non-existant concept to me before coming to UCSD, especially since most all of UW's speech/audio people resided in the EE dept, and few mingled with us. In general, it is also true that the computer vision field has flourished in the last two decades while its auditory counter part has not. Is there a bias for one sensory system over the other? Maybe. From my undergraduate perspective, any mention of the term "recognition" automatically applied to the computer vision field -- robots/machines that could "see" the way that humans do. It was always about hardware dogs that could play soccer, or search engines that could find pictures based on image content, or systems that could recognize hand gestures... the list goes on.

So when I was first introduced to the idea of computer audition -- engineering intelligence that can "hear" the way that humans do, I had one of those great "O_o" moments. Which is precisely what led me to UCSD in the first place.

I think my initial interest in computer audition was partly sentimental. Like many people, I'm crazy about music. Specifically, classical. I don't know much about signal processing or even music theory (sadly, even after many years of piano playing), but the simple fact that my auditory system has allowed me to enjoy so much music over the years seems to warrant at least some sort of return....as weird as that sounds.

Apart from being just really interesting, the field opens up a whole new world of possible technologies; many that can complement those that are being/have been developed in computer vision. If an intelligent machine has eyes, why not ears? These applications will have great educational, musical, informational, military, and medical impact; all of them inspired by aural abilities that most humans take completely for granted everyday. For example, we are currently thinking about possible technologies that may help the deaf or hard-of-hearing community.

The following is a set-up that gets right to the "heart" of what we wish to be able to do computationally. This is written by Albert Bregman in the classic book, Auditory Scene Analysis:

Imagine that you are on the edge of a lake and a friend challenges you to play a game. The game is this: Your friend digs two narrow channels up from the side of the lake. Each is a few feet long and a few inches wide and they are spaced a few feet apart. Halfway up each one, your friend stretched a handkerchief and fastens it to the sides of the channel. As waves reach the side of the lake, they travel up the channels and cause the two handkerchiefs to go into motion. You are allowed to look at the handkerchiefs and from their motions to answer a series of questions: How many boats are there on the lake and where are they? Which is the most powerful one? Which one is closer? Is the wind blowing? Has any large object been dropped suddenly into the lake?

Solving this problem seems impossible, but it is a strict analogy to the problem faced by our auditory systems. The lake represents the lake of air that surrounds us. The two channels are our two ear canals, and the handkerchiefs are our ear drums. The only information that the auditory system has available to it, or ever will have, is the vibrations of these two ear drums. Yet is seems to be able to answer questions very like the ones that were asked by the side of the lake: How many people are talking? Which one is louder, or closer? Is there a machine humming in the background? We are not surprised when our sense of hearing succeeds in answering these questions any more than we are when our eye, looking at the handkerchiefs, fails.