MF3D Background

Why are neuroscientists creating virtual reality monkeys? The answer requires a brief history of the field of social neuroscience and the importance of the macaque monkey as an animal model for understanding the human brain and psychiatric disorders.

Studying the social primate brain

While non-human primates (NHPs) make up only a very small fraction of animals used in biomedical research, the majority of NHPs used in research are Rhesus macaques. Data from NIH (2017).

In order to advance our limited understanding of how complex primate brains like ours work to coordinate social interactions, we need animal models that allow us to ethically study the neural correlates at the cellular level. Non-human primates represent a crucial resource in this endeavor, and the Rhesus monkey (Macaca mulatta) - a species of Old World monkey - is one of the most promising and popular (Figure 1; NIH, 2017).

Group-housed Rhesus macaques (image from CNPRC)

In the wild, Rhesus macaques have the second largest territorial range of any primate on earth (after humans). They live in large troops composed of up to hundreds of individuals, which are organized by strict social hierarchies and governed by rules of social interaction. They rely predominantly on their high acuity vision for gathering information about the intentions and internal states of their conspecifics. The similarity of these macaque traits to our own make the macaque the best available animal model for understanding the neural processes underlying complex social behaviors that are unique to primates like us.

Face processing in the macaque brain

Peri-stimulus time histograms (PSTHs) show changes in firing rate of a neuron in inferotemproal cortex that is sensitive to changes in the head angle of the macaque presented to the subject as photographic stimuli (Desimone et al., 1984).

Neural responses to images of faces in the macaque brain were first reported by Gross et al. (1972). The approach of recording extracellular spiking activity during visual stimulation of the anesthetized monkey followed that of Hubel & Wiesel (1968). Subsequent electrophysiology studies revealed that many of these neurons in the macaque superior temporal sulcus (STS) are sensitive to specific types of facial information, including head and eye gaze direction, identity, and facial expression (Perrett et al., 1982; 1985; Hasselmo, Rolls & Baylis, 1989).

Regions of ‘face selective’ voxels in the macaque brain localized using functional magnetic resonance imaging (fMRI) suggest a brain network (Landi & Freiwald, 2017).

The development of non-invasive functional magnetic resonance imaging (fMRI) in the 1990s lead to the discovery of face selective regions of ventral visual cortex in humans (Kanwisher et al., 1997) and subsequently the discovery that face-selective neurons were also clustered together in specific regions of the macaque inferotemporal (IT) cortex (Tsao et al., 2003; 2006; Pinsk et al., 2005), prefrontal cortex (Tsao et al., 2008) and medial temporal lobe (Landi & Freiwald, 2017).

Limitations of traditional vision neuroscience approach

Natural video stimuli used by Mosher and colleagues (2015) revealed that some amygdala neurons are sensitive to whether the subject is looking at the eyes of another animal. Click image to view movie.

The reductionist approach that has proven so successful for understanding the neurophysiological basis of visual processing in early visual areas, typically relies on simplified, abstract or artificial stimuli in order to allow for systematic, parameterized testing along stimulus dimensions of interest. However, the artificial nature of these experimental paradigms raises questions of their relevance to the types of visual experiences that the brain naturally encounters. This tension between rigorous control and ethological validity is perhaps most conspicuous in trying to understand the visual processing of social cues, to which much of the primate brain appears dedicated.

Consequently, scientists have more recently begun exploring alternative approaches that utilize more complex, ethologically valid visual stimulation paradigms (Hasson et al., 2003; Mosher, Zimmerman & Gothard, 2014; Russ & Leopold, 2015; McMahon et al., 2015; Park et al., 2017; Sliwa & Freiwarld, 2017). However, the trade-off of using ‘natural’ video stimuli is the loss of experimental control. While the visual and semantic content of video footage containing human actors is at least partially under the control of the director, natural movies of macaque subjects cannot be easily choreographed to address specific research questions.

Development of Macaque Avatars

Developmental history of digital macaque avatars. A. Steckenfinger & Ghazanfar, 2009 B. Ghazanfar et al., 2013 C. Artist: Tim Mrozek (Paukner et al., 2014; Simpson et al., 2016) D. Artist: Damien Montelliard (Wilson et al., 2019) E. (Siebert et al., 2020) F. Artist: Kelly Bullock, 2019 (Martinez-Trujillo lab, unpublished) G. Barbary macaque (M.sylvanus), by artist Shariq Altaf, 2018. H. Japanese macaque (M.fuscata) by artist Andre Holzmeister, 2017. I. Japanese macaque by studio Kleos Animation, 2018.

Digital macaque ‘avatars’ have been used in behavioral and neuroscience research for over a decade (Figure 1). This approach has typically involved a trade-off between ethological validity (i.e. how natural and/or ‘real’ the visual stimulus appears to the subject) and experimental control. Indeed, an early study concluded that macaque subjects may experience a perceptual phenomenon known as the ‘uncanny valley’, whereby semi-realistic artificial representations of faces elicit more negative emotional responses than either real or obviously artificial faces (Steckenfinger & Ghazanfar, 2009).

Since the earliest uses of macaque avatars (Figure 1A), technological advances in graphic processing power and decreases in cost have lead to increased accessibility and photorealistic quality of rendering for complex particle systems (such as hair and fur) and naturalistic lighting (such as subsurface scattering). Given a studio budget, state of the art CGI is now almost indistinguishable from video, while even low-budget projects utilizing open-source tools such as Blender can now achieve high-quality results.

Despite these technological advances, the time and cost of employing skilled digital professionals to generate realistic 3D models of macaques can be prohibitive for lab budgets. The duplication of these efforts across many labs who (understandably) choose not to freely distribute their intellectual property is an inefficient use of the research community’s collective resources. MF3D is the first ever publicly available macaque avatar resource for the scientific research community.

A note on ‘uncanniness’

Figure from Carp et al., 2022 illustrating the visual stimuli used to test the ‘uncanny valley’ hypothesis in macaque observers.

The ‘uncanny valley’ is a hypothesized psychological effect that postulates the relationship between an object’s resemblance to a ‘real’ animal (usually human) and the viewer’s emotional response to that object. Specifically, the hypothesis states that there is a proportional relationship between resemblance and positive emotional response, but a sudden dip (the ‘valley’) towards negative emotional response for objects that are almost life-like (such as a corpse, prosthesis, or animatronic robot).

The idea that macaque avatars may induce uncanniness in macaque observers was first explored by Steckenfinger & Ghazanfar (2009). By manipulating the resemblance of the visual stimuli to real animals (including both rendered and photographic images), and measuring subjects’ eye movements, the authors found that monkeys spent less time looking at face images of intermediate resemblance compared to photographs and unrealistic synthetic avatars. They concluded from this that their subjects did indeed experience an uncanny valley effect.

However, CGI and macaque avatars came a long way in the next decade and subsequent studies using newer avatars report conflicting results as to whether an uncanny valley effect actually exists in either humans or monkeys (Siebert et al., 2020; Wilson et al., 2020; Carp et al., 2022).

While the question of whether monkeys experience an uncanny valley type effect is important to researchers who seek to exploit the experimental control of virtual avatars in behavioral research with monkey subjects. However, for neuroscientists interested in how the macaque brain processes social information, the bar is much lower. ‘Face-selective’ neurons in macaque inferotemporal cortex will respond to a crude cartoon face composed of a circle with two eyes and a mouth, so it is clearly possible to address some questions about how the brain processes faces without the need for the subject to believe they are observing a real animal.

Virtual Reality: ethologically validity and experimental control

An example frame of the MF3D avatar inserted into a naturalistic virtual environment, rendered with Blender’s Cycles engine.

While the use of macaque avatars rendered in isolation is the logical extension of the traditional visual neuroscience approach, a major benefit of the flexibility provided by a digital avatar is the ability to embed it in more naturalistic virtual environments. This can take the form of either offline rendered film-like realistic scenes, or real-time rendered computer game-like scenes. These approaches offer the ability to address a range of research questions that it would not be possible to test using traditional video stimuli or real-life interactions between animals.