Sometimes we respond slow sometimes fast Residue Iteration

Sometimes we respond slow, sometimes fast Residue Iteration Decomposition (RIDE) Restoring latency-variable ERP components from single trials Intra-person & inter-person Variability: ² Difficulty for cognitive data analysis ² Opportunity for studying brain activity‐function relation RIDE: Team: 10/27/2020 v Un‐mix ERP components v Restore ERP component shapes v Get useful information about sub‐process variability in single trials Guang Ouyang Changsong Zhou Dept. Physics, Center for Nonlinear Studies, ICTS Hong Kong Baptist University Werner Sommer Department of Psychology Humboldt University Berlin, Germany 1

Outline v Motivation: Problems in ERP due to latency variability v Alternative model for ERP: Latency variable components v RIDE Ø Component decomposition Ø ERP reconstruction Ø Single trial variability Ø Comparison with ICA Ø Applications of RIDE in neuropsychology v Summary 10/27/2020 2

How do ERP researchers deal with “noisy” EEG signal? (b) 50 μV (a) EEG ERP average Stimulus onset ERP (Event‐related Potential) 10/27/2020 3

Averaging ERPs: Basic Idea 10/27/2020 4

ERP components as probes into cognitive stages Stimulus ERP-component 10/27/2020 Stage 1 Stage 2 C 1 Stage n C 2 Response Cn 5

Assumption of averaging ERP implying… … signal A B C … output signal A B C … output 0 ms 200 ms 400 ms 600 ms output 800 ms 1000 ms Identical processes in each realization; Contradictory to strong response variability 10/27/2020 6

Alternative assumption about brain response signal A signal … output B … C A B A signal A 0 ms C A signal B 200 ms C B … C B 400 ms output C 600 ms … 800 ms output … output 1000 ms Latency jitters in each realization 10/27/2020 7

Reality: systematic variability in the EEG data Sorting by RT What is the consequence of latency variability on ERP? 10/27/2020 8

Latency variability: Smearing effect (convolution) ØGenerally, the convolution reduces the amplitude ØWhen the ‘spreading’ distribution is broad, the convolution approaches to zero 10/27/2020 9

Stimulus Trial 1 0 ms 1800 ms Trial 2 Limitations of average ERP method Trial 3 0 ms Ø Smearing and mixing of components Ø Limited precision of amplitude/latency and conditional effects Trial i Ø Vague interpretation 0 ms in mental chronometry Ø Lose valuable single trial dynamic information A possible solution? . . . 1800 ms Average ERP 0 ms 10/27/2020 1800 ms 10

Stimulus Trial 1 0 ms 1800 ms Trial 2 Trial 3 Trial i . . . RIDE Ø Find latency-variable components 0 ms Ø Locate the latency in single trials Ø Reconstruct Average ERP latency-corrected ERP Reconstructed ERP Ø Improve understanding of brain-behavior relationship 0 ms 10/27/2020 1800 ms 11

Alternative model of ERP: temporal superposition signal A A signal B C B A Previous attempts: (with all time markers known) —Fourier decomposition … output … C B C output … output Mathematically they are the same! ✘ Cannot deal with latency-unknown components ✘ Suffer from strong low-frequency distortion Hansen, 1983 (low-frequency divergence) Takeda et al, 2008 (no asymmetry in R) —Iterative de-convolution Wordorff, 1993 Zhang, 1998 — General Linear Model decomposition (least square‐based) Dandekar, et al, 2012 10/27/2020 12

How does the single trials look like Response locked component cluster R Stimulus locked component cluster S 10/27/2020 Central component cluster C ‐ This is a single trial ERP data synchronized to stimulus onset and sorted by reaction time from a speech production data (from Oz). ‐ This is a very excellent data showing dissociated component clusters, but what if the components are serious overlapped? 13

How does the single trials look like Stimulus locked component cluster S ? ? Central component ? Response locked component cluster R cluster C ‐ This is a single trial ERP data synchronized to stimulus onset and sorted by reaction time from a speech production data (from Oz). 10/27/2020 14

RIDE: basic framwork 1. Initial estimation of latency of C 2. Decomposition Module Obtain S, C and R based on Ls, Lc and LR 3. Re-estimation of latency of C Remove S and R from single trials and estimate the latency of C by template matching between C and residue. convergence 10/27/2020 15

RIDE: decomposition module Latency for S Latency for C Latency for R Initially set S= C =R=0 Remove C and R from single trials get median of the residue for S Remove S and C from single trials get median of the residue for R 10/27/2020 Remove S and R from single trials get median of the residue for C 16

RIDE: working on components with unknown latency Stimulus onset Trial #1 S -- Stimulus-locked component Trial #2 C -- Non-time-locked component Trial #3 R -- Response-locked component … … Component #1 RIDE decomposition Component #2 Component #3 10/27/2020 Ouyang G. , Herzmann G. , Zhou C. & Sommer W. (2011), Psychophysiology, 48, 1631– 1647 17

RIDE: solving distortion problem Instead of using average in the iteration, we used median , 1983 Hansen 98 19 Zhang, . , 2008 l a t e a 2 Taked al. , 201 t e r a k Dande … Novelty of RIDE u Median based Numerical iteration (prevent distortion) u Estimate unknown latency 10/27/2020 18

RIDE: Typical assumptions in applications RIDE assumes: Ø Stimulus‐locked component cluster S Ø Latency‐variable central component cluster Ø Response‐locked component cluster R C S … Decomposition C Average ERP R Single trials This coincides with the assumption that the brain response goes through several stages, e. g. , Signal (Input) 10/27/2020 Perception Evaluation Response * RIDE is also Extendable to experiments without response markers: Nogo (S+C), Language (S+N 400+P 600) Behave (Click button) 19

RIDE Outcome: ERP component decomposition: Grand mean Consistency across subjects ERP C R S C *This figure is from the earliest RIDE algorithm, Ouyang et al. 2011 R 10/27/2020 20

RIDE Outcome: ERP re-construction--method Put at the most probable latency trial #1: component 1 trial #2: trial #3: component 2 . . . trial #n: component 3 Blurred ERP Decomposition Reconstructed ERP wave form: most probably observable in single trials 10/27/2020 21

RIDE Outcome: ERP re-construction: real data 10/27/2020 22

RIDE Outcome: ERP re-construction: conditional effects To solve the smearing effect due to latency variability Case I 1. Latency variability reduces the amplitude effect Cond. 1 Cond. 2 Case I 2. Different latency variability makes to amplitude effect 10/27/2020 23

RIDE Outcome: 10/27/2020 conditional effects in re-constructed ERP 24

RIDE Outcome: conditional effects in re-constructed ERP Rectified conditional effect in language processing Wang Fang et al. , 2015 10/27/2020 25

Does RIDE find artificial effect? - No 1. To test it, we randomly divided ERP dataset into two groups. In this case, there is not supposed to be significant conditional effects between the two groups. 2. We did this permutation for 100000 times, and did t test. 3. If RIDE finds more fake effects, the t distribution should be broader, as compared to standard ERP(see left). 4. However they are identical (see right) Standard ERP Re‐ERP t distribution 10/27/2020 26

Does RIDE find artificial effect? - No Another test: We generated 100000 set of data simulated by pink noise (1/f). 2 conditions and 20 subjects. In this simulation: 1. The number of finding p<0. 05 effect is 4981 out of 100000 (4. 981%) 2. The number of finding p<0. 01 effect is 956 out of 100000 (0. 956%) *Note: the two values above may change in other simulation but will be still extremely close to the threshold ‐ ‐ > From noise, RIDE does not find an effect with probability higher than chance. 10/27/2020 27

A clarification (usually misunderstood) A, B: RIDE only separates a C component when it already (visually) exists (i. e. , clearly visible). But due to the latency variability, its waveform is somewhat blurred and its amplitude is attenuated. RIDE can separate and resynchronize it to its own latency rather than stimulus onsets, and reconstructed ERP. C, D: RIDE does not create component from zero. If there is not visible C in single trials (C), or extremely weak, RIDE will not separate a C component for you (the C in the result will be like noise). Stimulus A Trial 1 Stimulus B Trial 1 Trial 2 Trial 3 Trial i standard ERP Re-constructed ERP 0 ms Stimulus C 10/27/2020 Stimulus D Trial 1 Trial 2 Trial 3 Trial i standard ERP Re-constructed ERP 0 ms 28

A clarification (usually misunderstood) Examples ‐ C is clearly visible in the standard ERP (usually a clear hump) C 2 C 1 ‐ If a ‘C’ is not even visible from standard ERP, then RIDE might not be able to extract it. RIDE restores component (from blurring), not creates components. 10/27/2020 29

RIDE Outcome: Single trial variability in latency and amplitude of each component 10/27/2020 30

RIDE vs. ICA (Independent Component Analysis) Illustration for two components’ case A Stimulus onset Latency jitter Trial #1 RIDE Component #1 Separation Trial #2 Component #2 Trial #3 … … B Component #1 Weighted Mixing ICA Component #2 10/27/2020 Component #1 Electrode #1 Unmixing Electrode #2 Component #2 31

RIDE vs. ICA (Independent Component Analysis) Some basic assumptions and features RIDE ICA Component number A few A lot Theory about different components Variable inter‐component delays across single trials Statistical independent from each other Basic Data requirement Multiple trials Multiple electrodes Linearity Linear 10/27/2020 (2, 3, 4…) (same as electrode no. ) 32

RIDE vs. ICA (Independent Component Analysis) A glance on the difference between RIDE and ICA in Real Application RIDE ICA 10/27/2020 33

What RIDE can do in neuropsychological applications? Examples, where RIDE can help with ERP analysis: 1. Desociate mixing of amplitude and latency effects 2. Desociate cognitive processes 3. Reconstruction of latency‐corrected ERP: N 400 and P 600 (Fang Wang) 4. Reject artifacts that is difficult to be rejected by other approaches (in preparation) 5. Intra‐subject variability (ISV) in individual difference and COMT effect (in preparation) 10/27/2020 34

Mixed condition effects on amplitude and latency 10/27/2020 35

1. Desociate mixing of amplitude and latency effects Nogo minus Go Difference wave Ouyang G. , Schacht A. , Zhou C. & Sommer W. (2013), Psychophysiology, 50, 253– 256 10/27/2020 36

1. Desociate mixing of amplitude and latency effects A parietal negativity due to the latency shift of a parietal positivity (C(+R)‐component) Go No‐Go Nogo‐Go 10/27/2020 Ouyang G. , Schacht A. , Zhou C. & Sommer W. (2013), Psychophysiology, 50, 253– 256 37

1. Desociate mixing of amplitude and latency effects From a face recognit ion data: Stimulus‐ locked Face priming effect can be separated into two distinct regions on the brain scalp 10/27/2020 Ouyang G. , Herzmann G. , Zhou C. & Sommer W. (2011), Psychophysiology, 48, 1631– 1647 38

2. Desociate cognitive processes Simon task =left =right Compatible Incompatible Problem: Onset time difference: 95 ms >> Reaction time difference: 43 ms After RIDE: 49 ms 10/27/2020 Stürmer B. , Ouyang G. , Zhou C. Boldt A. & Sommer W. （2013） Psychophysiology, 50, 70 -73. 39

Reconstruction of latency‐corrected ERP Rectified conditional effect in language processing 10/27/2020 Presentation by Fang Wang, SCN 40

Separate artifacts in Language Production Experiment: picture‐naming; R: VOT (voice onset time) word 1 word 2 word 3 word 4 word 5. . G. Ouyang, R. Abdel Rahman, S. Aristei, W. Sommer, and C. S. Zhou, in preparation 10/27/2020 41

Limitations ‐ The latency estimation of single trial latency (of C) introduces imprecision to the outcome The C latency is obtained by estimation based on cross correlation, which is certainly affected by noise. If the single trial noise is much stronger than C component itself, the precision of C latency would be compromised. In general it requires high signal to ratio of C component. Good C Noise summed slightly biased seriously distracted

Limitations C: [200, 900 ms] S: [0, 500 ms] ‐ Time window specification ERP The time windows for extracting different RIDE component is decided based on the observation of ERP (usually the grand mean). See the example on the right. R: RT± 300 ms S C R

Limitations ‐ Linearity assumption Different components are assumed to be linearly superimposed to each other. In reality, this assumption might be violated to different extents. ‐ Constant morphology (same limitation of ERP) RIDE component is approximation of the most probable morphology across single trials. But it does not represent the specific morphology for each single trial because the morphology may vary from trial to trial.

Summary: What RIDE can do? More applications of RIDE ERP components Decomposition/r econstruction Source Space Dynamical Causal Model 10/27/2020 Single trial ERP information/IS V Individual difference Development Aging Genotypes Diseases …. . Brain signal variability Resting/ongoi ng EEG 45

Summary-- RIDE: A New Paradigm of ERP analysis signal A … C 200 ms C B B … C B 400 ms output … C C 600 ms Ø Decomposed Component clusters Ø Reconstructed ERP Ø Single trial latency and amplitude 10/27/2020 output B A Outcomes: … B A signal 0 ms C A signal B … 800 ms output … output 1000 ms ² Toolbox ² Application Next talk… 46

Thank you for your attention! References for RIDE: 1. 2. 3. 4. 5. 6. 7. 8. 9. Ouyang, G. , Herzmann, G. , Zhou, C. , & Sommer, W. (2011). Residue iteration decomposition (RIDE): a new method to separate ERP components on the basis of latency variability in single trials. Psychophysiology, 48: 1631 ‐ 1647. Ouyang, G. , Schacht, A. , Zhou, C. , & Sommer, W. (2013). Overcoming limitations of the ERP method with Residue Iteration Decomposition (RIDE): A demonstration in go/no‐go experiments. Psychophysiology, 50: 253‐ 265. Stürmer, B. , Ouyang, G. , Zhou, C. , Boldt, A. , & Sommer, W. (2013). Separating stimulus‐driven and response‐related LRP components with Residue Iteration Decomposition (RIDE). Psychophysiology, 50: 70‐ 73. Verleger, R. , Metzner, M. F. , Ouyang, G. , Śmigasiewicz, K. , & Zhou, C (2014). Testing the stimulus‐to‐response bridging function of the oddball‐P 3 by delayed response signals and Residual Iteration Decomposition (RIDE), Neuro. Image, 100, 271‐ 280. Ouyang G, Sommer W, & Zhou C (2015) Updating and validating a new framework for restoring and analyzing latency‐variable ERP components from single trials with residue iteration decomposition (RIDE). Psychophysiology. In press. Ouyang, G. , Sommer, W. , & Zhou, C. A Toolbox for Residue Iteration Decomposition (RIDE) – A Method for the Decomposition, Reconstruction, and Single Trial Analysis of Event Related Potentials. Journal of Neuroscience Methods. In press. Wang, F. , Ouyang, G. , Zhou, C. S. , &Wang, S. P. (submitted). Re‐examination of Chinese semantic processing and syntactic processing: Evidence from Conventional ERP and Reconstructed ERP by Residue Iteration Decomposition (RIDE), PLo. S One, in press. Nowparast Rostami, H. , Ouyang, G. , Bayer, M. , Schacht, A. , Zhou, C. , & Sommer, W. Dissociating the Influence of Affective Word Content and Processing Demands on the Late Positive Potential. Submitted. Ouyang, G. , Abdel Rahman, R. , Aristei, S. , Sommer, W. , and Zhou, C. Speech Artifacts in Event‐Related Potentials: Description and Correction. In preparation 10/27/2020 47