Neuroscientific investigations of natural language processing, Olaf Hauk
Co-registering eye tracking with EEG and MEG
Olaf Hauk, Felix Dreyer, Maarten van Casteren, Elisabeth Fonteneau and Béla Weiss (University of Cambridge; Free University Berlin; Hungarian Academy of Sciences, Budapest)
Cognitive neuroscience research often employs simplistic and unnatural experimental paradigms. This is particularly the case in reading research, where stimuli are often presented word-by-word at a pre-specified location, while natural reading is a complex active process consisting of sequences of fixations. Unfortunately, studying natural reading with electro- and magnetoencephalography (EEG and MEG) is complicated by the presence of strong ocular artefacts, and the need to accurately co-register eye-tracking (ET) data with EEG/MEG. In this talk, we will start with a brief overview of approaches that have recently been introduced [e.g. 1,2]. We will present results from a combined EEG/MEG and eye-tracking study in which participants naturally read simple plausible and implausible sentences (“She ate the apple” vs “She ate the shoe”). ET and EEG/MEG data were recorded simultaneously. Behaviour was characterized by reading speed (RS), saccade amplitude (SA), fixation duration (FD), total number of saccades (TNS) and percentage of regressive saccades (PRS). EEG/MEG recordings were cleaned using independent component analysis (ICA). Source localization of cleaned EEG/MEG activity was carried out using L2 minimum-norm estimates. Behaviour before the target words was matched between conditions with respect to RS, SA, FD, TNS and PRS. Implausible sentences showed a more negative-going in the ERP response compared to implausible sentences in the N400 latency window. Cluster-based permutation testing resulted in a significant cluster comprising medial temporal, inferior frontal and supplementary motor brain regions in the left hemisphere, with stronger activations for plausible compared to implausible words. In conclusion, we found that our approach produces clean EEG/MEG data suitable for source estimation, that the well-known N400 effect can be replicated, and that plausible spatio-temporal brain dynamics can be revealed during natural reading.
Neural correlates of processing the perspective of other listeners during language comprehension
Shirley-Ann Rueschemeyer and Zdenko Kohut (University of York)
Humans are constantly engaged in social interactions, and many of these interactions are supported by language. Natural language use in communicative settings involves both understanding a language system, and understanding something about the social background of one’s conversational partners. In this talk, I will present the results of a recent neuroimaging study in which we demonstrate that listeners are attuned to background information about other listeners during language comprehension. Accessing social as well as linguistic information activates neural networks involved in both language processing and mentalizing. This is in line with previous studies showing an interaction between these networks during the processing of pragmatically difficult language stimuli. Interestingly, we show that connectivity between language and mentalizing networks is enhanced when the perspective of co-listeners diverges, providing insight into how these two high level cognitive systems work in concert to support social, communicative language processing.
Neural oscillatory mechanisms in dynamic information representation during natural audiovisual speech perception
Hyojin Park, Robin A.A. Ince, Philippe G. Schyns, Gregor Thut and Joachim Gross (University of Birmingham; University of Glasgow; University of Muenster, Germany)
Integration of multimodal sensory information is fundamental to many aspects of human behavior, but the neural mechanism underlying these processes remain mysterious. For example, during face-to-face communication we know that the brain integrates the dynamic auditory and visual inputs but we do not yet understand where and how such integration mechanisms support speech comprehension. Here we quantify representational interactions between dynamic audio and visual speech signals and show that different brain regions exhibit different types of representational interaction. With a novel information theoretic measure [1], we found that theta (3-7 Hz) oscillations in the posterior superior temporal gyrus/sulcus (pSTG/S) represent auditory and visual inputs redundantly (i.e. represent common features of the two) whereas the same oscillations in left motor and inferior temporal cortex represent the inputs synergistically (i.e. the instantaneous relationship between audio and visual inputs is also represented). Importantly, redundant coding in the left pSTG/S and synergistic coding in the left motor cortex predict behavior – i.e. speech comprehension performance. Our findings therefore demonstrate that processes classically described as integration can have different statistical properties and may reflect distinct mechanisms that occur in different brain regions to support audiovisual speech comprehension.
Neural mechanisms of natural speech comprehension in auditory and motor systems.
Anne Keitel, Christoph Kayser and Joachim Gross (University of Glasgow; Bielefeld University, Germany; University of Münster, Germany)
I am interested in intrinsic rhythmic activity in the human brain and how this might help us to understand natural spoken language. We have recently shown that each brain area has its own characteristic mix of intrinsic rhythms [1]. On the other hand, speech is also inherently (quasi-) rhythmic, with different rhythms for phonemes, syllables, words, and phrases. How our intrinsic brain rhythms can capitalise on speech rhythms to support speech comprehension is therefore a highly interesting question that can teach us more about mechanistic brain functions. Previous work that examined dynamic brain activity has addressed the issue of comprehension only indirectly, for example by contrasting intelligible speech with unintelligible speech. Recent work, however, suggests that brain areas can show similar stimulus-driven activity, but differently contribute to perception or comprehension. In this talk, I will focus on our recent study [2], which directly addressed the perceptual relevance of dynamic brain activity for natural speech encoding, by using a straightforward, single-trial comprehension measure. Furthermore, we based our analysis directly on the time-scales of phrases, words, syllables, and phonemes of our speech stimuli. By incorporating these two conceptual innovations, we demonstrate that distinct areas of the brain track acoustic information at the time scales of words and phrases. Moreover, our results show that the motor cortex uses a cross-frequency coupling mechanism, presumably to predict the timing of phrases in ongoing speech. Taken together, our recent findings suggest spatially and temporally distinct brain mechanisms that directly shape our comprehension, and that might be rooted in intrinsic brain rhythms.
Co-registering eye tracking with EEG and MEG
Olaf Hauk, Felix Dreyer, Maarten van Casteren, Elisabeth Fonteneau and Béla Weiss (University of Cambridge; Free University Berlin; Hungarian Academy of Sciences, Budapest)
Cognitive neuroscience research often employs simplistic and unnatural experimental paradigms. This is particularly the case in reading research, where stimuli are often presented word-by-word at a pre-specified location, while natural reading is a complex active process consisting of sequences of fixations. Unfortunately, studying natural reading with electro- and magnetoencephalography (EEG and MEG) is complicated by the presence of strong ocular artefacts, and the need to accurately co-register eye-tracking (ET) data with EEG/MEG. In this talk, we will start with a brief overview of approaches that have recently been introduced [e.g. 1,2]. We will present results from a combined EEG/MEG and eye-tracking study in which participants naturally read simple plausible and implausible sentences (“She ate the apple” vs “She ate the shoe”). ET and EEG/MEG data were recorded simultaneously. Behaviour was characterized by reading speed (RS), saccade amplitude (SA), fixation duration (FD), total number of saccades (TNS) and percentage of regressive saccades (PRS). EEG/MEG recordings were cleaned using independent component analysis (ICA). Source localization of cleaned EEG/MEG activity was carried out using L2 minimum-norm estimates. Behaviour before the target words was matched between conditions with respect to RS, SA, FD, TNS and PRS. Implausible sentences showed a more negative-going in the ERP response compared to implausible sentences in the N400 latency window. Cluster-based permutation testing resulted in a significant cluster comprising medial temporal, inferior frontal and supplementary motor brain regions in the left hemisphere, with stronger activations for plausible compared to implausible words. In conclusion, we found that our approach produces clean EEG/MEG data suitable for source estimation, that the well-known N400 effect can be replicated, and that plausible spatio-temporal brain dynamics can be revealed during natural reading.
- Dimigen, O., Sommer, W., Hohlfeld, A., Jacobs, A.M., & Kliegl, R. (2011). Coregistration of Eye Movements and EEG in Natural Reading: Analysis and Review. Journal of Experimental Psychology: General, 140, 552-572.
- Weiss, B., Knakker, B., & Vidnyánszky, Z. (2016). Visual processing during natural reading. Scientific Reports 6, 26902.
Neural correlates of processing the perspective of other listeners during language comprehension
Shirley-Ann Rueschemeyer and Zdenko Kohut (University of York)
Humans are constantly engaged in social interactions, and many of these interactions are supported by language. Natural language use in communicative settings involves both understanding a language system, and understanding something about the social background of one’s conversational partners. In this talk, I will present the results of a recent neuroimaging study in which we demonstrate that listeners are attuned to background information about other listeners during language comprehension. Accessing social as well as linguistic information activates neural networks involved in both language processing and mentalizing. This is in line with previous studies showing an interaction between these networks during the processing of pragmatically difficult language stimuli. Interestingly, we show that connectivity between language and mentalizing networks is enhanced when the perspective of co-listeners diverges, providing insight into how these two high level cognitive systems work in concert to support social, communicative language processing.
Neural oscillatory mechanisms in dynamic information representation during natural audiovisual speech perception
Hyojin Park, Robin A.A. Ince, Philippe G. Schyns, Gregor Thut and Joachim Gross (University of Birmingham; University of Glasgow; University of Muenster, Germany)
Integration of multimodal sensory information is fundamental to many aspects of human behavior, but the neural mechanism underlying these processes remain mysterious. For example, during face-to-face communication we know that the brain integrates the dynamic auditory and visual inputs but we do not yet understand where and how such integration mechanisms support speech comprehension. Here we quantify representational interactions between dynamic audio and visual speech signals and show that different brain regions exhibit different types of representational interaction. With a novel information theoretic measure [1], we found that theta (3-7 Hz) oscillations in the posterior superior temporal gyrus/sulcus (pSTG/S) represent auditory and visual inputs redundantly (i.e. represent common features of the two) whereas the same oscillations in left motor and inferior temporal cortex represent the inputs synergistically (i.e. the instantaneous relationship between audio and visual inputs is also represented). Importantly, redundant coding in the left pSTG/S and synergistic coding in the left motor cortex predict behavior – i.e. speech comprehension performance. Our findings therefore demonstrate that processes classically described as integration can have different statistical properties and may reflect distinct mechanisms that occur in different brain regions to support audiovisual speech comprehension.
- Ince RAA. (2017) Measuring multivariate redundant information with pointwise common change in surprisal. Entropy, 19(7):318.
Neural mechanisms of natural speech comprehension in auditory and motor systems.
Anne Keitel, Christoph Kayser and Joachim Gross (University of Glasgow; Bielefeld University, Germany; University of Münster, Germany)
I am interested in intrinsic rhythmic activity in the human brain and how this might help us to understand natural spoken language. We have recently shown that each brain area has its own characteristic mix of intrinsic rhythms [1]. On the other hand, speech is also inherently (quasi-) rhythmic, with different rhythms for phonemes, syllables, words, and phrases. How our intrinsic brain rhythms can capitalise on speech rhythms to support speech comprehension is therefore a highly interesting question that can teach us more about mechanistic brain functions. Previous work that examined dynamic brain activity has addressed the issue of comprehension only indirectly, for example by contrasting intelligible speech with unintelligible speech. Recent work, however, suggests that brain areas can show similar stimulus-driven activity, but differently contribute to perception or comprehension. In this talk, I will focus on our recent study [2], which directly addressed the perceptual relevance of dynamic brain activity for natural speech encoding, by using a straightforward, single-trial comprehension measure. Furthermore, we based our analysis directly on the time-scales of phrases, words, syllables, and phonemes of our speech stimuli. By incorporating these two conceptual innovations, we demonstrate that distinct areas of the brain track acoustic information at the time scales of words and phrases. Moreover, our results show that the motor cortex uses a cross-frequency coupling mechanism, presumably to predict the timing of phrases in ongoing speech. Taken together, our recent findings suggest spatially and temporally distinct brain mechanisms that directly shape our comprehension, and that might be rooted in intrinsic brain rhythms.
- Keitel, A., & Gross, J. (2016). Individual human brain areas can be identified from their characteristic spectral activation fingerprints. PLoS biology, 14(6), e1002498.
- Keitel, A., Gross, J., & Kayser, C. (2018). Perceptually relevant speech tracking in auditory and motor cortex reflects distinct linguistic features. PLoS biology, 16(3), e2004473.