Abstract
Our brain has incredible ability to receive sensory information from multiple modalities, integrate it, and use it to construct a best guess about what's going on around us. However, not every integration is beneficial. Discrepancy in the information carried by different senses may offset or cancel the multisensory benefit. I set out to investigate the costs and benefits of audio-visual integration with a video game platform. In the game, the visual signal is the rhythmic size oscillation of the game target whereas the auditory signal is an amplitude modulated sound that accompanies the oscillation in size. In the game, two species of fish were identical in appearance but differed in the rate at which their size fluctuates.The game requires players to discriminate between the two species of fish. The first study manipulated the coherence of temporal dynamics between the visual and auditory signals and examined the role of temporal coherence on multisensory integration. Study 1 showed a performance advantage rather than a cost associated with temporally matched visual and auditory signals. Study 2 Phase One identified the behavioral and EEG signatures of multisensory integration and found low alpha power and high theta power linked to the audio-visual incongruent condition. The changes seen in alpha and theta power reflects how the brain registers a mismatch between auditory and visual signals. In Study 2 Phase Two, game was modified to incorporate a mild stressor caused by increasingly reduced inter-fish interval, with the maximum allowed response time held constant. In addition to compromised task performance, the negative impact of reducing inter-trial interval was indexed by pulse rate and pulse rate variability. The result suggests perception of time pressure adversely affect the efficacy of cognitive decision making and the ability to adapt to changes. In summary, our brain and body are sensitive to pick up the discrepancies in the signals from different sensory channels. There will be a cost of cognitive flexibility with even the idea of time limitation.