Modelling fixational eye movements to achieve superresolution in deep neural networks

Abstract

Our eyes are constantly moving. Every second, we make three large saccadic movements, and in between these saccades, small fixational eye movements continuously occur. These eye movements are not random and serve crucial computational roles by focussing on relevant parts of the environment and allowing information to be integrated between eye movements. This active sampling of information is a hallmark of human visual processing but is currently difficult to model. Indeed, Deep Neural Networks (DNNs), the current state of the art for modelling the visual system, commonly lack eye movements and process static images in a single feedforward sweep. Understanding how to model eye movements and how to integrate this information over time is an important avenue of research. The following thesis focuses on modelling the small fixational eye movements that continuously occur between saccades. It has been shown that these fixational eye movements allow the visual system to reach superresolution to detect features of higher spatial frequency than what would be possible under static fixation. To model this process, we used a recurrent DNN combining supervised learning and deep reinforcement learning that can learn where to look in images. Reproducing the experiments conducted on humans, we trained the network to classify down-sampled high spatial frequency psychophysical stimuli that cannot be discriminated from the static image. We show that the network is able to learn useful fixational eye movements to achieve human-like superresolution on these stimuli and test to what extent this model can explain experimental data about human fixational eye movements. Finally, we show that this method can be applied to reach superresolution on naturalistic images.