The perceptual basis of knowledge representation
In this section, we return to the issue of how cognitive psychologists characterise conceptual structure. In particular, we return to the issue of simulations, which we introduced briefly in section 7.2.2, and attempt to see how these can be incorporated into a theory of frames. Of course, this relates to the more general question we have been pursuing in this chapter: what do the mental representations that underpin language ‘look like’? For cognitive linguists, the answer lies in the thesis of embodied cognition which gives concepts a funda mentally perceptual character. As Langacker argues, for instance, concepts are ultimately grounded in terms of basic domains which represent knowledge arising from foundational aspects of experience relating either to sensory experience of the external world or to subjective (or introspective) states. Our objective in this section, then, is to provide a sense of how the models of knowledge representation being developed in cognitive semantics are increasingly consonant with theories being developed in cognitive psychology. In particular, we address some of the more recent ideas that have been proposed by cognitive psychologist Lawrence Barsalou.
In his (1999) paper Perceptual Symbol Systems, Barsalou argues that there is a common representational system that underlies both perception (our ability to process sensory input from the external world and from internal body states such as consciousness or experience of pain) and cognition (our ability to make this experience accessible to the conceptual system by representing it as concepts, together with the information processing that operates over those concepts). One property of cognition that distinguishes it from perception is that cognition operates off-line. In other words, cognitive processing employs mental representations (concepts) that are stored in memory, and thereby frees itself from the process of experiencing a particular phenomenon every time that experience is accessed and manipulated. For instance, when planning a long car journey, we can predict roughly at what points in the journey we will need to stop and refuel. In other words, we can make predictions based on our concept – or frame – for CAR. We can make these predictions on the basis of past experiences, which come to form part of the mental representation associated with our mental knowledge of cars. This means we can make predictions about fuel consumption on a forthcoming journey rather than just getting into the car and waiting to see when the petrol runs out.
According to Barsalou, perceptual symbols (concepts) are neural representations stored in sensory-motor areas of the brain. He describes perceptual symbols as ‘records of the neural states that underlie perception. During perception, systems of neurons in sensory-motor regions of the brain capture information about perceived events in the environment and in the body’ (Barsalou 1999: 9). For example, consider the concept HAMMER. The perceptual symbol for this concept will consist of information relating to its shape, weight, texture, colour, size and so on, as well as sensory-motor patterns consistent with the experience of using a hammer (derived from our experience of banging a nail into a piece of wood, for example). It follows that perceptual symbols are multi-modal, drawing information from different sensory-perceptual and introspective (subjective) input ‘streams’.
However, perceptual symbols do not exist independently of one another. Instead, they are integrated into systems called simulators. A simulator is a mental representation that integrates and unifies related perceptual symbols (for example, all our experiences with hammers). Two kinds of information are extracted from simulators. The first is a frame, which we discussed earlier in the chapter (section 7.2.2). A frame is schematic in nature, abstracting across a range of different perceptual symbols for hammers. Hence, it provides a relatively stable representation (a concept) of HAMMER, drawing together what is uniform about our experience with tools of this kind.
The second kind of information extracted from a simulator is a simulation. A simulation is an ‘enactment’ of a series of perceptual experiences, although in attenuated (weakened) form. For instance, if we say ‘imagine you’re using a hammer . . .’, this utterance allows you to construct a simulation in which you can imagine the hammer, feel a sense of its weight and texture in your hand, and sense how you might swing it to strike another object. Therefore, part of our knowledge of the concept HAMMER includes a schematic frame relating to the kinds of knowledge we associate with hammers, as well as simulations that provide representations of our perceptual experience of hammers. Crucially, both frames and simulations derive from perceptual experience.
Evidence for the view that conceptual structure has a perceptual basis, and for the view that concepts (represented in terms of frames) can give rise to simulations, comes from a range of findings from neuroscience, the interdisciplinary study of brain function. This area of investigation has begun to provide support for the thesis that cognition is grounded in perceptual symbol systems of the kind proposed by Barsalou. For example, it is now clear that damage to parts of the brain responsible for particular kinds of perception also impairs our ability to think and talk about concepts that relate to those areas of perceptual experience. For example, damage to motor and somatosensory (touch) areas affects our ability to think about and identify conceptual categories like tools which relate to motor and somatosensory experience. Similarly, damage to areas of the brain that process visual perception affects our ability to access or manipulate conceptual categories that relate to visual experience. Evidence from experiments based on descriptive tasks also suggests that conceptual representation is perceptual in nature. For example, when a subject sitting in a lab without a perceptual stimulus is asked to describe a car, he or she will typically describe the car from a particular ‘perspective’: subjects tend not to list attributes in a random order, but to describe the parts of the car that are near each other first. Moreover, when a context is provided, this can influence the simulated perspective: subjects who are told to imagine that they are standing outside the car will describe different attributes of a car, and in a different order, compared with subjects who are told to imagine that they are sitting inside the car. This type of experiment suggests that the CAR frame, together with its associated simulations, is based on sensory-motor experience of cars.
Before concluding, let’s briefly compare models that assume a perceptual basis for mental representation with the type of model adopted in formal linguistics. Since the emergence of the Chomskyan mentalist model of language in the mid-twentieth century which firmly focused attention on language as a cognitive phenomenon and the simultaneous rise of cognitive science, theories of mental representation have adopted a non-perceptual view. This is sometimes called an amodal view, because it views conceptual structure as based not on perceptual (modal) states, but on a distinct kind of representational system. According to Barsalou, cognitive science was influenced in this respect by formalisms that emerged from branches of philosophy and mathematics (such as logic), and from the development of computer languages in computer science and artificial intelligence. Moreover, the prevalence of the modular theory of mind, not only in linguistics but also in cognitive psychology, represented a widespread view of perception and cognition as separable systems, operating according to different principles. This view is inherent in Fodor’s theory of mind, for example, which is outlined in his book The Modularity of Mind (1983). According to this theory, there are three distinct kinds of mental mechanisms: transducers (which receive ‘raw’ sensory-perceptual input and ‘translate’ it into a form that can be manipulated by the other cognitive systems), central systems (which do the ‘general’ cognitive work such as reasoning, inference and memory) and modules (specialised and encapsulated systems of knowledge that mediate between the transducers and the central systems).
In non-perceptual systems for mental representation, words assume primary importance as symbols for mental representations. For example, in early approaches to lexical semantics, feature lists employed words to stand for semantic features:
In formal semantics, the language of predicate calculus was adopted, which also based semantic features on words. While semanticists who rely upon componential and formal methods do not assume that words literally make up the content of the mental representations they stand for, they do rely upon items of natural language as a metalanguage for describing natural language, an approach that entails obvious difficulties. For example, if we rely on real words to express concepts, this limits the set of concepts to the set of real words. As we have seen, recent developments in cognitive psychology suggest that conceptual structure actually has a perceptual basis. These ideas, together with the empirical evidence that is beginning to be gathered, is consonant with the claims of cognitive semantics, particularly the thesis of embodied cognition.
