Our Senses: An Immersive Experience - Rob DeSalle 2018

Pani ca’ Meusa, Crème Brûlée, and Synesthesia: Crossmodal Impact on Taste and Synesthesia

“Crème Brûlée is the ultimate ’guy’ dessert. Make it and he’ll follow you anywhere.” —Ina Garten, cookbook author and host of The Barefoot Contessa

Taste is based on chemoreception, but it turns out that what we taste is heavily influenced by our other senses, such as sight and touch. On a visit to Palermo, Sicily, I had the pleasure of eating street food at the famous Ballarò open-air market. I stumbled on a vendor preparing and selling a Sicilian delicacy called pani ca’ meusa. Believe me, it did not look appetizing at all. That it seemed to be simmering in a laundry bucket made it look even more unpleasant. The color of the “delicacy” was a sickly gray with a greenish hue, and the texture of the meat in the sauce was, well, porous and spongy in places and rather jagged in others. I am being delicate here when I say it didn’t look edible, and when pani ca’ meusa was translated for me (bread with spleen), I was even more turned off. Then I learned that lung and throat cartilage were also elements of the sickly gray sauce, and I could make out the rough cartilage and even spongier lung particles in it. At this point I was massively second-guessing having ordered this so-called delicacy. My first taste of this street fast food, not surprisingly, gagged me a bit, because it was sour and had a lumpy texture. But I conducted a simple experiment. I closed my eyes, tried to put my preconceived notion of what a spongy gray food might taste like out of my mind, and took another bite. Not much better. I finished the pani ca’ meusa reluctantly. Later that week, though, after my preconceived notions had subsided, I tried it again. I can now say that pani ca’ meusa is perhaps one of the neatest, tastiest fast foods I have ever eaten and crave returning to Palermo’s street markets for more whenever I think of Sicily.

Part of the effect on perception of taste also depends on the semantics of description of the food. Actually, pani ca’ meusa has a pleasant-enough sound in Sicilian, so here is where I will momentarily drop the bread and spleen. Marketing researchers have started to use the intricacies of crossmodality to enhance sales of their products. Using experiments that suggest that a product’s name is incredibly important in how the product is perceived, marketing researchers have discovered some universals. One of the most amusing is that harder-to-pronounce wineries are perceived as making better-tasting wine than less-difficult-to-pronounce wines in head-to-head competition. So, the language used to describe food can influence taste greatly. Connect this factor with the tactile crossmodal interaction with taste, and we have a very complex way to perceive the taste of our food. We have seen that tactile information or texture of food is important to perception of taste. This also might mean that shapes are important in perception of food, and indeed research shows this very response.

Psychologist Charles Spence has made a career of understanding the crossmodality of taste with other senses and is described as a pioneer of taste studies. He has studied the speech sounds associated with chocolate, a great example of his interest in crossmodality and taste. Spence and his colleagues looked for any associations between chocolate and its cocoa content with rounded and sharp or angular words in different cultures, kind of like a taste-based Kiki and Bouba. They found that sweet or low-cocoa chocolates are almost always associated with round words with sound containing soft nasal and back vowels. Words such as “lula” and “maluma” are appropriate examples of these soft- or round-sounding words that are associated with sweetness. Bitter chocolate or high-cocoa-content chocolates are associated with words such as “tuki” and “takete.” Although the experiments they used clearly show that people can map sounds to tastes, the exact mode of the mapping is still unknown. But given Spence’s originality in experimenting and his interest in everything from the crunch of a Pringles potato chip to the fizzing sound of carbonated beverages, it would be a good bet that the crossmodal nature of a lot of these associations will be worked out soon.

Most of my original aversion to bread and spleen was visual. Specifically, warnings early in life from my parents never to eat grayish green meat probably contributed to my initial aversion to the delicacy. But another sense was probably at work as part of crossmodal interactions that caused my gag response. Researchers think there is also a tactile component to what we taste. Experiments where subjects are given food objects that are associated with specific tactile influence can address this possibility. In experiments, subjects were served the same food in two ways: with a rough surface and with a smooth surface. They also received the same food on rough or smooth platters. The researchers used the serving plate experiment as a control and indeed found that the texture of the serving plate had no effect on gustatory reception. The texture of the surface of the food, however, had a significant impact on what study subjects were tasting. Whenever a person tasted the food with a rough surface, he or she perceived it as sourer than the same food served as smooth. Perhaps the rough texture of the throat cartilage mixed into pani ca’ meusa induced a similar sour taste.

The next question is: Does language based on shape act in a crossmodal way with taste? The answer is yes. In clever experiments, researchers determined that the semantic space of taste and shapes contains two principal clusters. This is just a fancy statistical way of saying that their experiments revealed two associations of words and shapes. One cluster consisted of the word “sweet” and round shapes. The other cluster included shapes that had edges and the taste words “salty,” “sour,” and “bitter.” Using these associations, the researchers timed study participants’ response to incongruent and congruent pairings. This test is kind of like a shape/taste Stroop experiment (box 15.1). In this Stroop taste test, a series of food items, each having a specific shape and a specific taste, is presented to the person being tested. So, for instance, the person might be given a series like the following: a round/sweet piece of food, then a jagged/sour piece, and finally a triangular/bitter piece. The subject is asked to name the tastes in the series and is timed doing the task. More shape and taste combinations follow, such as star shape/sweet, followed by oval/sour, followed by round/bitter. The subject is again asked to identify the tastes. When the shape and taste combinations are incongruent—for example, star shape/sweet—it takes longer to identify the taste correctly. This result indicates that there is a crossmodal connection of taste and shape as mediated through language.

BOX 15.1 | THE STROOP TEST

The Stroop test is a classic vision-semantic test that pairs words with color in a unique way. It consists of two lists of words in which the words are printed in different colors. In one list the letters are colored the same color as the word, so BLUE would be blue in color, GREEN would be green in color, and so on. The second list mixes up the colors and the words, so that BLUE would be colored red, GREEN would be colored purple, and so on. The person taking the test is asked to verbally list the colors of the words. One can easily and quickly list the colors of the words when the name of the color is matched with its visual color. But to call out the colors of the words on the list correctly when the colors don’t match the word takes more effort and much longer. The Stroop test is a classic example of a “team of rivals” effect. When we see the word RED colored blue, this produces a conflict in our brains that requires sorting out, which takes some time. When we see the word RED colored red, there is no conflict, and hence we can quickly make the decision that the color of the word is red.

Words are obviously part of these crossmodal interactions. But what about other sounds we make, such as music? One need look no further than the music in TV commercials to conclude that music might be importantly linked to our perception of taste. It could be argued that there is some aesthetic component to the choice of music in a TV commercial, if one can imply that a commercial has aesthetics. But there might also be some very distinct crossmodal interactions going on in such choice. Now we can go one step further and determine whether music is involved. Enter Charles Spence once again. With several colleagues, he has examined the potential role of music in perception of taste. In a series of experiments, Spence and his colleagues showed that sweet and sour tastes are associated with high pitches, whereas bitter tastes are linked to low pitches. Even musical instruments elicit connections to tastes. For instance, piano sounds are linked to sweetness, and trombones map to bitter and sour tastes.

These results simply ask what taste words are associated with basic units of sound or music. Digging deeper into the crossmodal role of music, Spence and colleagues looked at musical composition as a potential player in crossmodality. They asked a sound-branding agency to create four musical pieces that varied in auditory pitch, sharpness, roughness, and discontinuity. Using what is called a forced choice experiment, these scientists then asked participants to assign each of four taste words to the four compositions. The forced choice aspect of the experiment was that they were instructed to match one of the taste words with one of the compositions so that there were four unique matches. The researchers also assumed from their first experiment that there were correct assignments, such that sweet would be paired with a higher-pitched, softer, more continuous, and less sharp-sounding composition. And the other taste words would be associated with appropriately manipulated compositions. The results indicate that sweetness can be matched to its “correct” composition significantly better than randomly, with the salty taste word to its correct composition slightly less accurately, and the bitter and sour taste words least accurately. Both of these experiments support a hedonic model of crossmodality. In other words, the pairing of sweetness with more continuous and less sharp music can be based on the pleasure factor of the taste and the sound. In general, moderate sweetness is more pleasurable to the brain than bitterness, sourness, or saltiness. Likewise, more continuous and less sharp music is more pleasurable to people than discontinuous, sharp-sounding music. The overlap in pleasure level from each might be driving the capacity to pair sweetness with the more pleasurableness of the music.

Another experiment Spence and colleagues designed was relatively complex, because it paired four tastes with six kinds of music, for twenty-four possible combinations of music with taste. The six kinds of music involved subtle changes using attack, discontinuity, pitch, roughness, sharpness, and speed. For instance, pitch could be varied by the subject on a scale they controlled from high pitch to low pitch. The subjects were shown a single specific taste word (“sweet,” “sour,” “bitter,” or “salty”) and then asked to find where on the sliding scale for the six aspects of music the word could be matched. So, for instance, for the word “salty” the subject would read the word and would find the pitch that matched the word. Next, they would see the word “salty” and then find where on a sliding scale speed matched the word, and so on, for all four taste words and all six aspects of music. Surprisingly, all but attack of the music showed significant mapping with taste words. And in some cases, musical aspect can rank the taste words. For instance, auditory or musical roughness was significantly correlated to taste words with “sour” being associated with very rough sounds, “salty” with medium rough sounds, and “sweet” with the softest sounds. In general, sweet is mapped with higher pitch, softer sound, more continuous sound, lower tempo, and less auditorily sharp sound. Sour, on the other hand, can be paired with low pitch, soft sound, discontinuous sound, high tempo, and sharp sound.

The final experiment Spence and his colleagues designed considered the possible cultural differences that might be involved in the crossmodality. They recruited subjects from the United States and India and repeated the forced choice experiment with the four musical compositions and four taste words. The decision to contrast Indian subjects with American subjects was based on the cultural context of music preferred in the two countries. Indian music is microtonal; it uses intervals smaller than a single semitone. Western music as preferred by people from the United States uses a specific twelve-semitone interval system. This may not sound like much when read on the page, but the differences in music caused by these two profound structural preferences ensures quite different musical perception, preference, and processing. The results of the experiments indicate that American subjects are better at matching the “correct” composition with the taste word, suggesting that a cultural component is involved in the crossmodality. Besides telling us something about crossmodality, these experiments have an obvious marketing function. They point to what the Internet calls the now popular “pretty stupid yet insanely cool” Häagen-Dazs app. This app plays a classical concerto as a timer as you wait for your ice cream to breathe before eating it. It appears that Häagen-Dazs could have chosen the concertos for their ice creams a little more judiciously if they had followed Spence and colleagues’ advice on how to pair music with sweet taste. The concertos on the app are apparently not optimized for pairing sweetness with composition characteristics. On the other hand, rap music might be a good way to advertise salty products because of the pairing of discontinuous choppy music with salty taste words. And most important, if you are an Indian marketer, you wouldn’t want to pair a classical concerto with the sugary dessert rasgulla in TV advertisements.

Smell can also be crossmodal with sound and even music. Again Charles Spence, the guru of this kind of work, and his colleagues have studied the pairing of smell with sound. To test the validity of the olfactory-music crossmodality, these researchers used previously demonstrated olfactory-included crossmodalities. For instance, it is well known that smells are paired with shapes. Spence and colleagues were able to pair such smells as crème brûlée more with rounded shapes than with musky odors. Using the strengths of these crossmodalities, they then assessed the strength of any olfactory-music crossmodalities. In their first set of experiments, sweet, round, and higher-pitched correspondences were detected. As with the taste and music results, the response is more than likely hedonic, since sweet tastes, roundness, and higher pitch are all associated with pleasantness. A second set of experiments paired musical compositions with three smells: candied orange, crème brûlée, and ginger cookies. Most people in this study easily paired the candied orange smell with its “correct” music. On the other hand, people would randomly pair crème brûlée with all three musical compositions. And very strangely, ginger cookies were never matched to the “correct” composition. These results are revealing of a crossmodality of olfaction and sound, and specifically with music, but as the response to ginger cookies indicates, the interactions are most likely very complex.

Thus far we have examined ten of the fifteen possible binary crossmodal interactions for six senses (sound and sight, touch and sight, touch and sound, balance and vision, taste and sight, taste and sound, taste and touch, smell and touch, smell and sight, and smell and sound). We are missing four of the balance-with-other-sense pairs and one other pair: smell and taste. The four balance pairs that are missing would indeed be difficult to show existed, but I would not be surprised if Charles Spence figures out a way to pin these down if he puts his mind to it. The remaining pair is smell and taste. These two senses are well known to be intricately bound together, and their crossmodality is not in doubt. We have also looked at some three-sense crossmodalities, such as taste, sound, and sight. But the message should be that a team of rivals is taking the sensory information generated by our sense organs and interpreting it.

Figure 15.1. Nigel Tufnel’s famous amplifier dial that “goes to eleven” and is “one louder.”

Before we leave crossmodality, I want to return to Nigel Tufnel, the lead guitarist for the mythical heavy metal band Spinal Tap, mentioned in Chapter 7. His famous guitar amplifiers all had volume dials that went to eleven (fig. 15.1). When asked why not just number the dial levels from one to ten and have ten as the loudest, he responded, “Well, it’s one louder, isn’t it? It’s not ten. You see, most . . . most blokes, you know, will be playing at ten. You’re on ten on your guitar . . . where can you go from there? Where? Nowhere. Exactly. What we do is if we need that extra . . . push over the cliff . . . you know what we do? Eleven. Exactly. One louder.” There is actually some crossmodal sense to Tufnel’s bantering, as demonstrated by the following experiment. Subjects were exposed to a reference sound and asked to compare it to a second sound of either louder or softer magnitude. A small number (one, two, or three) or a large number (seven, eight, or nine) accompanied the second sound, and the subject was asked whether the reference sound was louder or softer than the second sound. The trick of the experiment is to expose the number to the subject simultaneously with the second sound and after a pause between the sound and the exposure to the number. When the number and sound occur simultaneously and sounds are paired with large digits, the subjects perceived the sound as louder than when sounds are paired with small digits. By separating the sound from the number though, the effect disappears. In this case seven, eight, and nine are indeed louder than numbers smaller than them, and if that is the case, then eleven could be louder than ten, too.

I have glossed over one important point about all of these experiments on crossmodality. Synesthetes were excluded from all of them. These phenomena are shown to exist in individuals without obvious connections of senses, as synesthetes clearly show. And synesthesia has added a great deal to our understanding of crossmodality and, more important, to the route and mode by which action potentials originating from our sense organs (eyes, ears, or noses) traverse our brains. Synesthesia, like the crossmodal interactions I have just discussed, comes in many varieties. According to many synesthesia experts, there are probably between 65 and 150 kinds of synesthesia. Perhaps the most accurate estimate is 80, as listed by Sean Day. To cover all 80 would be a book in itself. So I will touch on several subjects where some of the 80 types can help us understand something about the senses and, more important, help us uncover something about how our brains work to create perception and, ultimately, consciousness.

Each kind of synesthesia is made up of an inducer and a concurrent. The inducer is comparable to the trigger for the synesthetic experience. Without it, the synesthetic response cannot occur. If it is present, then the inducer does what its name suggests and induces a sensory response that is not a part of the common response specific to the inducer. The concurrent is the sense or sensory response that results from the presence of the inducer. For instance, the most common synesthetic response is where numbers or letters (also known as graphemes) act as inducers and the concurrent is color. Over 60 percent of people with clear synesthetic abilities have this grapheme-color pairing. Indeed, the clearest way to determine if someone is synesthetic is to test for this pair. It also turns out that if someone is synesthetic for a particular inducer-concurrent pair, he or she might very well be synesthetic for another or even several other pairs.

Synesthesia was at first thought to be more prevalent in females. The ratio of female to male synesthetes differs depending on which expert is consulted but ranges from six to one all the way down to one to one, where there is no female bias in the trait. Sean Day suggests that synesthetes make up about 4 percent of the human population. But this number will vary, too, depending on the study and researcher. Part of the difficulty in pinning down these frequencies of synesthesia in human populations is that you simply can’t use self-reported synesthesia in making the estimates. Consequently, several clever tests of whether a synesthetic claim is genuine have been developed. A modified Stroop test is one of the basic tools used to detect synesthesia. The only problem with the Stroop test is that some nonsynesthetes can train themselves to overcome the propensity to use the word they are seeing to describe the color and therefore appear to be synesthetic. Some individuals have also trained themselves either purposefully or simply by life exposure to inducer-concurrent pairings and hence can “cheat” their way to being considered synesthetes. The most visible of such false synesthetes are people who remember the colors of refrigerator magnet letters from childhood and retain the association of the magnet color with the alphabet letters. Children’s alphabet books can elicit the same false grapheme-color synesthetic response.

The most widely used test for synesthesia is the test of genuineness (TOG) and its improvement, the revised test of genuineness (TOG-R). These tests rely on the ability of the potential synesthete to systematically repeat the synesthetic response when challenged at different times. So, for instance, the potential grapheme-color synesthete is presented with a long list of graphemes that include words, days of the week, numbers, and letters of the alphabet. After the mention of each grapheme, the subject is asked to describe their precept of the grapheme. Different TOGs will use different inducers, but the real meat of the test lies in the repeating of the graphemes at different points. The TOG-R is a more complex and more precise method of testing for synesthesia that has also been cross-validated by applying the test to a large sample of known synesthetes. The Stroop test, mentioned earlier, has been suggested as a potential way to detect synesthesia. Remember that this test causes confusion when nonsynesthetes attempt to assess color in conflicting situations. When the nonsynesthete is challenged by words like “blue” printed in colors other than blue, such as green, they will confuse the color of the word and use the word itself, in this case “blue,” to describe something that is green. Synesthetes will in general not be fooled by the Stroop test.

Synesthesia has been studied by scientists for about two hundred years. The first study of synesthesia, published in the 1820s, happens to have been conducted by a synesthete whose dissertation was in part a study of himself. The phenomenon piqued the interest of researchers but didn’t really get the bump it needed until Francis Galton, Charles Darwin’s cousin, first looked at the psychometrics and possible genetics of synesthesia in the late 1800s.

Galton was a famous figure in Victorian scientific circles. He was a polymath who was difficult to categorize by his specialty and a brilliant statistician, given the mathematical tools of the time. Psychologists often times claim him as one of their own because he invented many of the early statistical tools that were used in the science of psychometry popular at the time. Synesthetes were of particular interest to Galton as a psychologist, and he was the first to show that synesthesia ran in families. He also showed clearly that not all synesthetes see the same color associations when challenged with identical graphemes. He also championed genetics as a scientific discipline. Unfortunately, he also defined and championed the now-disavowed pseudoscience called eugenics. Although Galton gave eugenics its name, he didn’t name synesthesia, even though the phenomenon was a subject of great interest to him. Rather, that distinction belongs to Jules Millet, who named the phenomenon in 1892. Erica Fretwell, an expert on late nineteenth-century literature and culture, has pointed out that synesthesia and eugenics were bound together, partly because of Galton, but also as a result of contemporary social mores. Victorian intellectuals were hung up on the differences within our species. Synesthesia offered yet another set of criteria for characterizing differences, and because it was neural or brain based, it linked more than genetics to these differences.

Binding synesthesia to eugenics was an unfortunate association for synesthesia. But since the demise of eugenics, the study of synesthesia has opened up many avenues of research on neural processes, and there should be no guilt by association of synesthesia with eugenics. Nevertheless, studies of synesthesia waned in the early twentieth century when eugenics went out of vogue. Synesthesia studies luckily reemerged around the late 1980s. The focus on synesthesia in a genetic context has led to several advances in how we view the phenomenon. Galton noted the familial correspondence of synesthesia but lacked the tools to pin down its genetic basis. Also, the sex bias that has often times been associated with synesthesia has misled researchers for some time regarding the location of genes involved in synesthesia. If the sex bias is as high as some studies suggest, then this might mean the trait is linked to the X chromosome, much like the X-linked tetrachromatic vision characteristics discussed in Chapter 3. The early observation Galton made about synesthesia running in families has also come into play in trying to uncover the genetic basis of synesthesia. A form of synesthesia called color-sequence synesthesia was examined in a twin study in 2015 by researchers Hannah Bosley and David Eagleman.

For this form of synesthesia monozygotic (identical) twins have a heritability of the trait that is about 74 percent, and dizygotic (fraternal) twins have a heritability of 36 percent. If the trait were controlled entirely by genetics, then the monozygotic twins would have 100 percent heritability. If the trait were not inherited, then it should appear in either kind of twin at a much lower concordance. The 74 percent heritability of the dizygotic twins in this study suggests that this kind of synesthesia has a genetic component, but it is not total, and that expression of the trait has a considerable environmental component. For another type of synesthesia, the genome-wide association study approach is useful.

Modern genomic studies using the genome-wide association study approach exploit whole genome sequences of reference populations and the sequences of individuals with a trait of interest. These genomes are mined for positions that vary among the controls and the people with the trait, called a single-nucleotide polymorphism (SNP; see Chapter 8). When the human germline cell chromosomes replicate to make sperm and eggs, recombination, or genetic exchange of information, between the chromosomes in cells inherited from the parents occurs. These replication events break up the sequential arrangements of single-nucleotide polymorphisms with traits depending on how close the trait is to that polymorphism. If the polymorphism and the trait are very close to each other on a chromosome, then the recombination or shuffling of the trait and the polymorphism will be infrequent and the trait is said to be linked to it. Pinning down an association therefore requires showing that a specific polymorphism or multiple polymorphisms are linked to the trait. Furthermore, if we can determine where the polymorphism is on a chromosome, it is possible to associate particular genes in the region of the polymorphism with the trait.

Genome-wide association studies suggest that one kind of synesthesia (straight grapheme-color synesthesia) can be mapped to four of the twenty-three human chromosomes (none of these are the sex chromosomes, X and Y) and another kind (colored sequence synesthesia) can be mapped to chromosome 16. Of the four chromosomes in the first case that show linkage to synesthesia, chromosome 2 is the most significant. The other three chromosomes (5, 6, and 12) show suggestive linkage. To show the potentials and pitfalls of the approach, consider some of the genes near the linked single-nucleotide polymorphisms on chromosome 2 that associate with straight grapheme-color synesthesia. Looking for genes associated with a trait is a little like looking for a needle in a haystack, as we discussed with James Lupski and CMT syndrome in Chapter 8. If an association gets made, then the haystack gets reduced a lot, but some luck is still involved. Fortunately, 80 percent of the twenty thousand or so genes in the human genome have known, precisely described functions. The location of where a gene is expressed, how the protein translated from the gene works in development and in regular physiology, what the protein does in pathways, and other parameters are well known for these characterized genes. So, what kind of genes should we be looking for near the single-nucleotide polymorphisms that are associated to the trait? One obvious category would be any gene involved in how our nervous system works or genes that affect the development of our nervous system. Another category might be genes linked to neurological disorders or other neurological anomalies.

It turns out that the polymorphisms linked to synesthesia on chromosome 2 are in the same region as genes linked to autism. People with autism experience sensory-related abnormalities, and synesthesia is often a secondary feature of autism spectrum individuals. Indeed, people with autism spectrum disorder appear to have an increased frequency of synesthesia. Finally, using functional magnetic resonance imaging (fMRI), researchers have shown that auditory stimuli will excite similar auditory and visual regions of the brain for both people with autism and people who are synesthetes. As far as candidate genes go, a gene linked to synesthesia on chromosome 2 called TBR1 is involved in telling other nervous system genes when to express themselves. In other words, TBR1 regulates several genes important in neural development, including a gene named reelin (a gene involved in cerebral cortex development). Another gene involved in neural processes that is also found in the region of the genome with the associated polymorphisms is called SCN1A. This gene encodes a protein that lies in the membrane of synapses and is involved in processing action potentials across synapses. People who have altered forms of this gene suffer from epileptic seizures. Circling back to autism, researchers also know that rare variants in TBR1 and SCN1A are found in people with autism. Chromosome 16, mentioned earlier, also has single-nucleotide polymorphisms associated with colored sequence synesthesia. This kind of synesthesia, which is triggered by sequences of graphemes such as ABCD to produce colors, is quite different from grapheme-color synesthesia, so it is not surprising that it might be found on different chromosomes than the grapheme-color type. Six genes in this region are involved in the development and maintenance of the nervous system in the cerebral cortex. But when these genes were examined closely for variation between synesthetes and nonsynesthetes, no variation could be correlated to the trait.

As I was writing this book, I took my two-year-old son to a hearing specialist for hearing tests. He had just had tubes put into his ears to facilitate drainage of fluids that impaired his hearing. I was a little skeptical about the whole process of testing hearing in a two-year-old. How would they get him to indicate which ear a sound is coming from? One side effect of his hearing impairment is delayed speech, and he certainly couldn’t communicate verbally the way I did during my last hearing test. Because he is a typical two-year-old, I was asked to hold him during the test. I was simply amazed by how precisely the testers could interpret his head and eye movements to indicate where he heard the test sounds coming from. Researchers using similar approaches can now test children as young as two months for various sensory perceptions. You’d think that these tests would be rather simple, but it takes clever thinking to get into the mind of a two-month-old. Of course, infants don’t know what letters or numbers are, but researchers can actually do tests with the two-month-old-baby version of grapheme-color synesthesia. The test is based on the idea that a two-month-old baby can associate shapes with colors. So, for instance, if the baby associates a triangle with a red color, the baby will not clearly distinguish a triangle set on a red background but will see the triangle clearly on a green background.

The trick of the test, as with the hearing test on my two-year-old son, is to read the reaction of the two-month-old who is exposed to these figures. It turns out that a baby will stare at something interesting, and viewing figures like triangles and squares is more interesting to a baby than looking at a continuous red background. So, if the baby associates a triangle with red and a triangle is presented with a red background on the left and a green background on the right, the figure on the right, where the triangle is visible, will be more interesting and he or she will stare at it exclusively. Switching sides of the two figures should result in the baby staring to the left. Almost all babies stare nonrandomly at these figures, and they do so across trials on the same day as well as trials on different days. Two interesting secondary results from such experiments add to how a baby’s responses inform us about synesthesia. First, babies who pick up the red-green contrasts grasp them much earlier in life and the yellow and blue associations a little later in development. And second, as the baby grows, the synesthetic effect tends to diminish.

In the aggregate, these data reveal that most, if not all, babies are synesthetic just after birth. This observation is called the neonatal synesthesia hypothesis. The reason babies observe red-green associations before yellow-blue ones is that in the brain the red-green connections develop first and the yellow-blue channels emerge later. The final observation of diminishing synesthesia in growing toddlers, requires that we understand something about how infant and young brains develop.

Brain development has two main phases. Early in development we are born with almost all of the neural cells we will need in life. Studies that examine the neural connections of these cells indicate they are almost indiscriminately connected to each other by producing connections in an incredibly large number through synapses. Such synaptic connections cross sensory regions of the brain and in essence connect the different senses. The second phase occurs as the baby develops and is characterized by a process known as pruning. The neural connections that implement the universal neonatal synesthesia are pruned away, leaving only neural connections that process the visual information exclusively. Other neuroanatomical and neurophysiological data support this way of looking at the neonatal synesthesia hypothesis.

Another synesthetic system, colored hearing (where specific pitches of sound are paired with specific colors or shapes), can also be examined in infants. As we saw with our discussion of crossmodality, higher pitches are usually associated with Kiki-like sharp, pointy shapes and lower pitches are paired with Bouba-like rounded shapes. With this kind of synesthesia, high pitches are also paired with smaller, “higher in space” shapes and lower pitches with larger, “lower in space” shapes. In this case, detecting the baby’s response is critical, and it turns out that the test can be done because babies like to look at things. So, if a baby is shown a visual in which a shape morphs from being Bouba-like (amoeboid) to Kiki-like (sharp and prickly) and the sound accompanying the visual goes from a low pitch to a high pitch, the baby’s response should be different than when the visual is the same but the sounds go from high pitched to lower pitch. Specifically, because the Bouba-to-Kiki visual spectrum paired with the low-pitch-to-high-pitch sound spectrum is congruent and the same Bouba-to-Kiki visual paired with high pitch to low pitch is incongruent, the baby will be more attentive and watch the visual longer for the first combination than the second. Another test of this kind of synesthesia in infants is accomplished by showing a baby a brightly colored ball moving up and down in space. Paired with the ball’s movement are sounds going either from low to high tones or high to low tones. Again, if auditory synesthesia is at work, the baby should prefer the pairing of a ball in the down (up) position with a low (high) pitch to the nonconcordant opposite pairings. All babies as young as three to four months indeed pay more attention to the congruent visual-audio pairing than to the incongruent one, suggesting that infants are synesthetic with respect to pitch-shape and pitch-height pairings.

As I mentioned earlier, there are more than eighty kinds of crossmodal synesthesia. Going into each one would be a repetitive task, so in the rest of this chapter we will look at specific aspects of synesthesia that help us better understand the senses.

First, let’s consider studies that connect synesthesia with brain anatomy. Brain imaging studies can do a couple of things. For one, they can decipher the location in the brain where specific sensory information is processed. And more important, they can lead to an explanation for why synesthesia occurs in the first place. As inferred from the infant synesthesia studies, proximity of neural connections is a factor in promoting synesthesia in an individual. In infants, the connections are made but later pruned away. In true synesthetes, the connections are not pruned away and remain into later life. The earliest brain imaging studies of synesthetes were accomplished in the 1980s by infusing radioactive xenon with oxygen, which was then inhaled by a subject who was being evaluated for synesthesia. Xenon is an inert gas and is not harmful to a person’s physiology, but it can be traced using the same methods used to detect X-rays, so the subject wore a cap with an array of X-ray detector devices to measure where xenon (and hence oxygen) went in the brain during synesthetic activity. During these tests the subject breathed in oxygen and xenon, put on the detector cap, and then underwent exposure to a synesthetic inducer. Although the method could detect increased brain activity, it did so only rather crudely. These experiments (using state-of-the-art techniques and knowledge from more than thirty years ago) demonstrated that the cortex experiences increased activity, but learning how was not more precisely feasible.

With the invention of positron emission tomography (a functional imaging technique used in nuclear medicine to reveal metabolic processes in the body), MRI, and later DTI (which provides information about the brain’s white matter, or axons), researchers have been able to measure brain activity during synesthetic experiences more precisely. Grapheme-color synesthesia, because it is one of the more common forms of true synesthesia, has been the focus of detailed hypothesis testing about how the phenomenon works. Researchers can accomplish this most efficiently by using imaging, specifically fMRI, which highlights regions activated by a specific function. In essence, the brain is painted in different hues according to the level of activity caused by action or functioning as a result of a stimulus. The first fMRI studies of synesthetes claimed a clear correlation of grapheme-color synesthetic activity with regions of the optic pathway that correspond with the function of processing color. Later fMRI studies were designed to test hypotheses about the nature of synesthetic activity, as contrasted with the simple brain-painting experiments that were initially accomplished.

One of the more popular and logical hypotheses that can be tested concerns the process of cross-activation, which predicts a different pattern of brain activity than other models of synesthesia. Cross-activation would produce brain activity in two different parts of the brain, one where the inducer signal is being processed and the second in the region where the concurrent gets activated. The most recently developed approach to be used to examine synesthesia is DTI, and as of 2014, fewer than seven studies had been conducted using this approach. As discussed in Chapter 4, this approach can pinpoint neural connections that are active during specific neural activity. Early results using both fMRI and DTI point to a possible importance of localized brain function and connectivity, but unfortunately, there is so much variation in synesthetic activity even among grapheme-color synesthetes that interpretation of the data is difficult.

Jean-Michel Hupé and Michel Dojat performed a meta-analysis of all of the published synesthesia brain imaging studies in the literature in 2015. Their conclusions are nicely summed up in the following statement from their work: “We did not find any clear-cut empirical evidence so far about the neural correlates of the subjective experience of synesthesia. We did not find any structural or functional anomaly in the brain of synesthetes that could explain synesthesia. In our view, most published studies to date show, in fact, that the brains of synesthetes are functionally and structurally similar to the brains of nonsynesthetes.” In essence, synesthesia is very complex, and perhaps we need a new theoretical way of thinking about how the various forms of it work in conjunction with how the brain is structured and functions. Like an interesting phenomenon in nature, we always have more to learn, and synesthesia appears to be an enigma waiting to be cracked.

We very likely have more than the five Aristotelian senses plus balance on which I have focused thus far. But is it possible that the big six themselves are actually more than just six? There are multiple brain regions where the pathways for each sense traverse. Vision, for example, processes all kinds of aspects of that sense, including color, shade, orientation, movement, and others. In many ways, the most common synesthetic phenomenon, grapheme-color synesthesia, is really not characteristic of almost all other synesthesias. It is what is called an intramodal synesthesia because you need to see to visualize letters and numbers, and this requirement then induces another visual concurrence. Hearing is another, and recent synesthetic research suggests that specific aspects of hearing can be atomized. A rather famous hearing illusion, called the Doppler illusion, discovered in the 1990s, illustrates how the sense of hearing is rather tightly wired with different components—one for loudness and one for pitch.

The Doppler illusion occurs when one hears a sound that is continuously increasing in intensity over time. It can occur while you are standing still and the sound is increased at a constant rate, or it can happen if you are standing still and the sound is constant but approaching you at a constant rate. What happens when one is exposed to this kind of sound is that the tone’s pitch appears to increase as the loudness of the sound increases. In this case, the pitch actually isn’t increasing, but the loudness is. These two aspects of auditory sensing—loudness and pitch—compete for the right to interpret the sound. The solution is to let the loudness dictate the concurrent response in pitch. This intramodal synesthesia lies in strong contrast to all of the other eighty intermodal synesthesias and suggests that hearing at least might really be two senses.

Two final synesthetic oddities concern ideas that may eventually transcend how we normally look at the phenomenon. Both are somewhat controversial and are based on much smaller sample sizes than the better-known forms of synesthesia such as grapheme-color synesthesia. The first oddity concerns a neural phenomenon that is really in a category by itself. Twenty years ago, Giacomo Rizzolatti and his colleagues noticed something interesting about the macaques they were studying. They were evaluating neural signals in the brains of these primates as they reached for food or other objects, and they recorded the electrophysiology of single neural cells during these actions. Oddly enough, when the macaques observed a human or another monkey reaching for an object, the very same neural cells showed activity. The neurons involved in this strange phenomenon were given the name mirror neurons. Other researchers later established that there are mirror neurons not only for movement of the hands but also for facial and mouth gestures. Mirror neurons have since been used in the neurobiological community to better understand many cognitive mechanisms, such as how organisms understand goals and how organisms behave empathetically. They have also been used in studies of autism. But the most recent work on the mirror neuron phenomenon has been linked to certain kinds of synesthesia, too. In mirror touch synesthesia, individuals who see someone else being touched actually feel as if they are being touched. So this involves a visual induction, followed by a concurrent tactile sensation.

Finally, the literature contains several reports of emotional responses associated with sensations. The evidence for this synesthetic oddity is sparse, however. In one subject, certain textures elicited very specific emotional responses, such as denim being associated with depression. In the common grapheme-color association, some people showing this association will also associate emotions with the grapheme-color response. In 2013, researchers treated an individual who suffered a stroke that caused a lesion in the thalamus. This lesion resulted in the first reported case of acquired synesthesia involving emotion and grapheme-color synesthesia, because the person affected reported feeling disgusted when he read words printed in blue and less so when reading words in yellow. Even more interesting was this subject’s response to brass musical instruments (specifically, the James Bond movie theme) that the subject described as orgasmic. Both of these last phenomena are quite strange indeed. As if our brains weren’t klugey enough in dealing with duality, crossmodality, and synesthesia, we also now can see how our brains are wired to invite our emotions and memories and other aspects of higher cognitive processes into interpreting our outer world. How we invite the emotions and memories into the game of perception is incredibly revealing to how our brains make sense of the world.