1 Background

1.1 General context

We define brain decoding as the task of predicting mental states from brain activity: in short, one trains a model predicting what a person is seeing, hearing or thinking from recordings of their brain activity.

While brain decoding is not a new task, major advances were made in the past 2-3 years thanks to the advent of (1) large datasets collected in a small number of participants and (2) the availability of state-of-the-art pre-trained AI models allowing to obtain rich embeddings of the mental states at hand. In particular, recent pieces of work used functional MRI to successfully decode seen images [Ozcelik and VanRullen, 2023; Shen et al., 2019], heard speech [Jalouzot et al., 2024; Tang et al., 2023] and watched videos [Thual et al., 2023, 2024]. Interestingly, other studies have successfully relied on Magnetoencephalography (MEG) [Benchetrit et al., 2024; Défossez et al., 2022] to decode similarly complex mental states.

1.2 Modern approach to training brain decoders

Pieces of work previously cited in this document rely on a similar method to acquire MRI data used to train models decoding brain activity. Participants are scanned while being exposed to controlled stimuli – for instance images or sentences – which we refer to as samples. Samples would typically be shown at a rate of 1 every 5 seconds. Moreover, each sample is associated with a vector, which we refer to as an embedding or a latent representation. This embedding is obtained by passing the sample through a pre-trained deep learning model – for example CLIP [Radford et al., 2021] for images. Finally, brain decoding aims to train a model to predict these embeddings from recorded brain activity. Note that if the embedding predicted by the model is good enough, one can leverage it to reconstruct the image seen / sentence heard by the participant by using it to condition a generative AI model.

2 Objectives

Brain decoders trained on functional MRI data typically rely on heavily pre-processed data. Indeed, magnetic resonance is used to reconstruct images of the brain, and in particular track the oxygenation level of the blood in the cortex. However, the raw data acquired by an MRI scanner is not not an image, but a time series of complex numbers lying in the spatial frequency domain, which we refer to as k-space. This k-space data is then transformed into an image using the inverse Fourier transform, which is a lossy operation. We seek to train and evaluate brain decoders using raw k-space data as input rather than reconstructed images.

During this internship, we expect the intern to lead / participate in the development of the following deliverables:

• Internal documentation of a grid-search evaluating the performance of decoders trained using different setups (e.g. different learning rates, output embeddings, etc.)

• A public Python git repo containing the code used to train and evaluate the decoders •

• A submission to a conference or journal presenting the results of the project

  To best integrate the intern into the team, we expect them to regularly engage in peercoding sessions with their close team members. They will also interact closely with their mentor to prevent roadblocks and ensure steady progress throughout the internship.

We are looking for candidates with the following qualifications:

• A strong background in Machine Learning and Deep Learning

• Proficiency in Python and ML libraries such as PyTorch or TensorFlow

• Independence and ability to efficiently collaborate with other team members

Useful but not mandatory qualifications include:

• Experience with biosignal processing

• Interest in neuroscience

• Introduction call with Head of TA (Paul) - 30min

• Interview with Project lead (Alexis) - 45min

• Technical interview (onsite) - 1h30