Our Senses: An Immersive Experience - Rob DeSalle 2018

Team of Rivals Meets the Kluge: Making Sense Out of Crossmodal Stimuli from the Outer World

“And the blind man said to the deaf man, ’Do you see what I hear?’” —Wayne Gerard Trotman, filmmaker

Catchy names for our brains and how they work are aplenty. Two of my favorites are “kluge” and the “team of rivals.” I have come up with a few more, which I will not mention here out of embarrassment, but I like these two monikers because they encompass the evolutionary, psychological, and neurological context of our brain. The word “kluge” as a descriptor of the brain comes from neurobiologist Gary Marcus and his book of the same name. Marcus describes our brains as functional Rube Goldberg machines using the German word kluge, defined as “an ill-assorted collection of parts assembled to fulfill a particular purpose.” “Team of rivals” comes from David Eagleman, also a neurobiologist, from his book Incognito. His way of describing our brains plays off the well-known strategy Abraham Lincoln used in assembling his cabinet during the Civil War (as made famous by historian Doris Kearns Goodwin).

On the one hand, the brain has this messy makeup that defies most architectural or engineering logic. Some have likened it to a computer, but this analogy is mostly incorrect because computers have not evolved the way brains have. It is true that computers have changed over time, and it could be argued that they have evolved, but brains have had a much more checkered history than computers. There are no mulligans or do-overs in evolution, so the structures we see in organisms, and especially in brains, are the result of common ancestry. Hence brains are molded by the historical contingency of the evolutionary process. In fact, the best and most innovative solutions in computing are probably those that scrap a good deal of previous work and start with a relatively clean slate—a computer mulligan.

The human brain is a pretty bad example of good engineering, but it works. Once a structure or behavior arises as a product of mutation and is amplified as a result of natural selection, or a structure or behavior arises and is amplified by genetic drift, no matter how klugey, it is retained in a population. And if natural selection acts further to increase the frequency of the klugey solution, a population cannot simply scrap the solution for a less klugey approach. Quite the contrary, natural selection is forced to use the variation that exists, and if the variation is klugey, then chances are that the product of natural selection will be even more of a kluge. There are cases where the phenotype of individuals in a population change significantly and rapidly to give a product of natural selection pretty different from the original variants (by genetic drift or other evolutionary-developmental processes called hopeful monsters), but for the most part, the variants that are there are what natural selection works with. This doesn’t mean that things, once klugey, will always get klugier. But it does mean that kluginess can beget even more kluginess, and this is more than likely what happened with brains, as exemplified by following the changes throughout the tree of life and especially in the vertebrate part of the tree of life.

Eagleman’s team of rivals analogy is also a wonderful descriptor of the neurological context of the brain and how we sense the outer world. According to historian Doris Kearns Goodwin, Abraham Lincoln’s strategy for filling his cabinet in the 1860s was to enlist people who he knew would conflict with him and each other and would thereby produce more than conflict for the sake of argument. These rivals worked well together despite their ethical dilemma over slavery, conflicting political views, the threat of secession, and the ravages of the Civil War. The brain takes in all the contradictory information from the outer world gathered by our senses, the stimulation of chemicals from our nervous and hormonal systems, our physiology, and other extraneous and rival information and interprets it to keep us functioning. Conflict and rivalry then become important aspects of how our brains work. If there is no conflict, then there is no problem, and the operation is completed without a hitch or hiccup. For instance, Eagleman uses a car turning a corner as an analogy of something that is not conflicted and, hence, no problem. A steering wheel and the driver who turns that wheel control a car. A car does not complain about what it is doing when turning, and hence there is no conflict. The brain does not work that way. How we turn outer world stimuli into perception can be looked at and analyzed from two directions: bottom up and top down.

The brain is continually exposed to conflict from rival signals and information. The problem is to decipher how we process the information. Psychologists seem to think that we can do this in two general ways. If the information coming into the brain is optical in nature, then it takes a circuitous route in the brain to determine shade, color, shape, and other aspects of what we have seen. Memory and emotions kick in to give an overall perception of what our eyes just saw. Because this way of perceiving starts with data or information and the information is built upon by other neural functions with increasing complexity, psychologists call this a bottom-up approach to perception. The other approach to perception starts with the sensory information triggering memories of and emotions about things we have seen and interacted with in the past. A database, so to speak, exists in the brain that reacts to initial stimuli. Using this experiential reference, our senses then track the information through other parts of the brain to construct a perception. This route is pretty much the opposite of bottom up and is called the top-down approach to perception. The latter sorting of information (top down) to create perception is triggered in the context of memory, emotion, and other higher-order functions of the brain.

The importance of neural connections in how we use the information from our senses and integrate the information from multiple sources to formulate our perception of the outer world is a matter of integration. But is it top down or bottom up? In fact, in many ways it doesn’t matter which it is, and perhaps it is even a mixture of the two approaches to perception that acts in our brains. Just realizing that these are two possible processes allows psychologists to formulate testable hypotheses and create clever experiments to test the hypotheses about how perception works. Although this approach focuses on being able to reject hypotheses, in this experimental framework often we cannot reject anything in the context of reality. But what comes out of the experiments in this kind of scientific method is both greater understanding of how the brain processes information and more refined hypotheses of perception.

The neural pathways in the brain that take information from the outer world through the sense organs and travel through the brain are often circuitous. The routes through the brain for each of the big six senses vary, indicating that different parts of the brain are responsible for processing different sensory information. We have already seen how the sensory cortex is involved in processing touch, and although we can map the pathways to their processing points by means of homunculi, there are other parts of the brain where signals from the sense of touch must interact with memory and other higher functions. Perhaps the best-understood sense for the intricate pathways that are involved in interpreting sensation is sight (fig. 13.1). Years of neuroanatomical and psychological work have pinned down pretty precisely the major pathways in the brain for vision.

Vision starts within the eye and the neural impulses that are created by light hitting the rods and cones in the retina. The pathway that the impulses travel from the retina to the brain is decipherable simply by tracing the anatomical structures emanating from the eye. The nerve cells coming from the eye are bundled into rather large neural structures, the optic nerves. The optic nerves from the two eyes cross from the left eye to the right side of the brain and from the right eye to the left side of the brain via the optic chiasma. Just past the optic chiasma the two nerve bundles reach farther into the brain and connect to structures called lateral geniculate nuclei, one on each side of the brain. The two lateral geniculate nuclei then serve as relay stations that send impulses farther to specific regions in the back of the brain, where the optic nerve cells start to radiate. The discovery of the functions of the radiations or streams of neurons is discussed in detail below. The processing of the information doesn’t stop with these radiations or streams that reach into the visual cortex, but then loops out to the prefrontal cortex, where the information is placed into working memory so that we can easily access the information for further use. What happens in each of these brain areas has been the subject of much research and has revealed a great deal about how complex the brain really is.

Figure 13.1. Neural pathways for vision (top), hearing (bottom left), and smell (bottom right).

A century ago, using the clinico-anatomical correlation method, several German neuroanatomists pinned down that a specific region of the brain, when injured, would result in a visual anomaly called agnosia. Their observations allowed them to generalize that people with severe injuries in two regions of the brain—the lower region of the temporal lobes and the ventral occipital cortex—were unable to identify objects placed in front of them. More specifically, two gyri (the convex rolls of the human brain)—the lingual gyrus and the fusiform gyrus—were most often the site of brain lesions causing visual agnosia. Since the eye itself is not the source of the cause of visual agnosia, the actual visual information from the outer world is not deterred from entering the brain. Rather, the damage to these specific areas of the brain prevents processing of the information and results in the agnosia. In some rather brutal experiments, two neurobiologists in the 1930s removed chunks of rhesus macaque (Macaca mulatta) brains to determine the impact of lost function from these regions. Heinrich Klüver and Paul Bucy conducted the now famous and Hannibal Lecter—like experiments that bear their name: Klüver-Bucy syndrome. Although the removal of a big chunk from one side of the brain can often be overcome by rewiring, Klüver and Bucy did what is called a bilateral removal. In other words, they removed the corresponding chunks from both sides of the brain. In so doing, Klüver and Bucy produced monkeys that were very messed up with respect to vision, which made them very messed up in other behavioral aspects. They were unable to correctly or even slightly recognize images, and this lack of visual capacity severely altered several behaviors dependent on image recognition, such as eating and sex.

Brutal as these experiments might be, and indeed they would probably be condemned today by many animal rights advocates, they did lead to an explication of the neural pathways involved in vision. It is safe to say, though, that without these macaque brain ablation studies, we would have had to rely on the vagaries of the clinico-anatomical correlation method and more than likely would have only a very partial picture of the pathways involved in vision. The most important work from these ablation studies revealed two pathways through which action potentials involved in sight are processed. It turns out that there is an upper (dorsal) pathway and a lower (ventral) pathway of neurons that process visual information. The ventral stream takes care of the “what” of objects in perception, and the dorsal stream takes care of the “where” of objects in perception (box 13.1).

Within each stream neural impulses take well-defined pathways, and hence each part of the brain in these streams does well-defined tasks (fig. 13.2). For example, how does the brain process color?

BOX 13.1 | WHAT AND WHERE

Heinrich Klüver and Paul Bucy used ablation studies in macaques to show that disruption of the ventral pathway (or in macaque brain anatomy terminology, the occipito-temporal stream) by removal of chunks of the lower temporal lobe resulted in monkeys who cannot discriminate among objects. These were essentially monkeys who probably mistook their mates for a banana. Although these monkeys had trouble with figuring out “what” things were that they saw, they retained spatial visual acuity and could easily judge perspective and distance of objects (“where”). Ablations in the dorsal pathway (the monkey anatomy term is occipito-parietal stream) produced monkeys who could identify objects but had huge difficulty with spatial vision, or the “where” of objects. These results led to the understanding of the “what-and-where” dichotomy of the ventral and dorsal streams of visual information.

Color is part of the “what” that gets processed in the brain in the ventral pathway, so the color-processing part of the brain is in the ventral stream in the temporal lobe. In addition to color, this part of the brain also processes shapes, shades, and textures, except that these other aspects of “what” are processed in their own subareas of the ventral stream called “V” areas. These ventral processing functions are distinct and relegated to very specific regions of the brain and are thought to be sequentially connected. In fact, the V regions are numbered, and their numbering reflects their place in a spatial hierarchy of each of the two streams.

Figure 13.2. The visual “what” and “where” neural pathways in the human brain in the visual cortex.

What about something that is moving? Here we want to know about “where” the thing that is moving can be placed in space. So, this kind of information is “where” information, and as we said earlier, “where” is processed in the dorsal stream visual pathway. Indeed, things like motion, direction, and speed of objects are processed in the dorsal stream and again in unique V regions of the brain. The route through which action potentials move can be straightforward or circuitous depending on the difficulty of pinning down the “what” or “where” of the object. In other words, although researchers have understood the spatial location of the V regions and other regions of the temporal and parietal lobes important in processing visual stimuli, the route the neural impulses take is not only nonlinear but also nondirectional. The neurons in the different areas of the brain where the information is processed have multidirectional function. This means that vision is not necessarily unidirectional from neurons deciphering less complex perceptions to neurons solving more complex perceptions. The system is best described as having massive feedback and feedforward functionality. In addition, some linkages skip hierarchical levels in the ventral and dorsal streams. And to make the situation even more complex, there are potential connections within the individual V regions of the visual pathways that are critical for processing visual stimuli.

How does this neural architecture fit with top-down and bottom-up processing of visual stimuli? It means that both types of processing are possible. Anything with a feedforward neural pathway would be part of a bottom-up process, and anything that might feed back would be part of a top-down process. And it is not hard to visualize neural responses to visual stimuli that bounce around a bit with both feedback and feedforward patterns, making some of the processing of a cohesive visual stimuli a top-down process and other parts of it a bottom-up process.

The neural pathways for the big five senses are fairly well known, and the general theme of the route through which the initial action potentials from the external sense organs travel follows the overlying thematic tenet of sensory pathways architecture I call “It’s complicated.” Smell and taste receptors are triggered chemically. The action potentials produced by chemoreception in the nose have a relatively short distance to travel, because the olfactory bulbs that are the first stop for the impulses lie almost directly above the receptors for smell. The impulses take this somewhat direct route to the brain and pass through the olfactory bulbs, where primary processing is accomplished, and on to the primary olfactory pathway (see fig. 13.1). This cortex then passes neural impulses to the hypothalamus and thalamus of the so-called limbic system in the interior of the brain and also to the orbitofrontal cortex in the frontal lobes of the brain. The latter location is responsible for decision-making. All of these connections more than likely evolved as a means for rapid decision-making of organisms based on olfaction.

The taste receptors in the papillae of the tongue bind to the chemicals of food or beverage we put in our mouths. Such binding then sends action potential to the brain by the cranial nerves. Although these pathways are not as direct to specific areas of the brain as for smell, they reach some of the same regions of the brain that olfactory impulses do. Three major cranial nerves take the neural impulses generated by taste receptors from the front two-thirds of the tongue, and a single cranial nerve transmits the information from the throat, the top of the mouth, and the back third of the tongue to deep into three areas of the limbic system, which includes the thalamus. From the limbic system the impulses are transferred back out to the gustatory cortex, where the source of the impulses is interpreted as sweet, sour, salty, bitter, umami, or some combination. The gustatory cortex is located in the orbitofrontal cortex, where olfaction is processed. This pathway partially explains why taste and smell are so closely coordinated. In fact, what we call taste is really a multisensory experience, combining smell, taste, and texture. Many researchers argue that this close integration of these three sensory pathways is the result of extreme natural selection on populations of organisms to make quick and precise decisions about the things they put in their mouths. Of course, as the senses are interpreted in the brain, this information then interacts intricately with the physiology of the organism with respect to the reward system of the brain. The mouths of organisms have evolved to be pretty sophisticated sensing organs.

Hearing starts with the intricate structures in the inner ear that produce action potential as a result of sound waves hitting the intricate device of the inner ear (Chapter 5). The rest of the system is every bit as complex as in vision (see fig. 13.1). A complete description would be impossible in a paragraph, so I simplify it here a bit. After impulses are created as a response to sound waves, they travel to a group of nerve cells called the organ of Corti and also by means of one of the cranial nerves that wires the inner ear to the brain stem, where a connection with a group of nerve cells called the cochlear nuclei is made. In addition, there are connections to the thalamus in the limbic system. There is one more specific connection, and that is to the primary auditory cortex of the brain into what is called the superior temporal gyrus (one of the convex rolls of the brain in the temporal lobe). The impulses travel to many parts of the brain for higher-order processing, such as understanding language and responding to language, as is the case for Broca’s and Wernicke’s regions of the brain.

Because the vestibular system interacts with very basic movements of the body to maintain balance, and a huge number of muscles are used to do this, the pathways for this sense are very complex, too. The cerebellum ultimately controls balance, so the axonal makeup of this pathway fords its way to this structure at the base of the brain. To accommodate this system there are what are called ascending pathways (taking information up the spinal cord to the cerebellum) and descending pathways (taking information from the brain stem back down the spinal cord). The process starts when one of the major cranial nerves carries the initial action potential from the inner ear to the base of the brain. Once there, the impulses go to various clumps or nuclei of nerve cells in the medulla and pons of the brain stem and in the cerebellum. The various clumps of neural cells are responsible for different aspects of balance. The nuclei in the pons and medulla connect to descending pathways. One pathway that lies laterally connects to the spinal cord and traverses the length of the spinal cord. Walking upright in a balanced fashion is an outcome of proper signaling along this pathway. Another lies medially, uses the spinal cord to travel to midthoracic regions of the spinal cord, and both controls how we move our head and balances how our eyes and heads move.

The five or more kinds of touch sense receptors in the skin (Chapter 8) are stimulated mechanically and produce action potentials that need to travel to the brain. These impulses ultimately traverse the brain to the sensory cortex, where the touch impulses are interpreted and acted on. There are three major highways to the brain from these sensory organs embedded in the skin based on what the information conveys. Touch and the sense of where our bodies are in three-dimensional space travel to the brain by means of neurons that run along the back (dorsal) side of our bodies. Major aspects of proprioceptive stimuli that are dealt with by our sense of balance also travel to the brain via the spinal cord. Impulses essential in sensing temperature and pain travel in yet a third pathway to the brain. Once the impulses get to the brain, they congregate in the primary somatosensory cortex, that region of the brain we discussed extensively in the context of homunculi in Chapter 3. Most of the rest of the story of the senses is about what is called multisensory integration, or crossmodal interactions. These interactions are important for fast, accurate, and sometimes lifesaving interpretation of sensory information.

That smell you noticed or that flash of light or the breeze hitting your arm are all complex perceptions that are processed in your brain. None of these—the overall perception of smell or sight or touch—is actually only a single sense at work but usually the product of senses interacting.

Consider touch in this context. The information gathered by our touch neural cells is sent by action potential from the various kinds of cells in our skin that detect the touch. This action potential is then integrated by different parts of the brain. The sensory cortex is intricately involved, as shown through Wilder Penfield’s surveys of people during brain surgery. But more than just the sensory cortex processes touch. Our brains could easily stop at processing the touch information without complicating matters, but in a world of natural selection and genetic drift more complex things happen. To attain maximum resolution of the tactile stimulus that is important in an adaptive context, our brains incorporate more information about the touch for our species’s survival. With respect to touch, it is well known that with it, brain activity is enhanced not only in the somatosensory cortex but in other places in the brain. The activation is in areas of the brain responsible for seeing and hearing, among others. The reason for this is that the signal our brains are trying to perceive from the initial touch might not be pure. By pure I mean that the stimulus might not have the right level of information for the brain to reach a reasonable conclusion about the tactile event. The original touch might be a hard collision of the skin with an object, and in this case the information going to the brain might be so chaotic that it overwhelms the brain and causes problems with the interpretation of that original touch. More likely, though, the original touch may be so light that other senses are needed to amplify the information that goes to the brain. Indeed, crossmodality is usually most important in situations where the original sensation is very weak, suppressed, or broken down. The brain still needs to interpret the signal in some way. A good starting point is to return to thinking about neural rivalries, and these exist for nearly all of the senses.