[Article] MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding & Generative language reconstruction from brain recordings & CorText-AMA: brain-language fusion as a new tool for probing visually evoked brain responses
MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding
Summary: In this seminar, we will explore a novel fMRI-to-text decoding framework named MindLLM. It combines a neuroscience-informed, subject-agnostic fMRI encoder with an off-the-shelf large language model to translate brain activity into coherent text. It introduces Brain Instruction Tuning (BIT), which enriches the model’s capacity to extract and represent diverse semantic information from fMRI signals, enabling versatile decoding across different tasks and subjects. We will discuss how to implement these techniques in our study.
Generative language reconstruction from brain recordings
Summary: Using non-invasive fMRI, the authors map brain-decoded semantic representations into a large language model so it can autoregressively generate text aligned with the perceived stimulus—achieving direct brain-to-language generation without pre-constructed candidates and better alignment than selection-based baselines.
CorText-AMA: brain-language fusion as a new tool for probing visually evoked brain responses
Summary: CorText-AMA introduces an end-to-end brain–language framework that fuses fMRI signals with a large language model to caption and answer questions about natural scenes via an interactive chat interface, enabling targeted probing of what visual-cortex activity encodes and outperforming control models using functional alignment.