Interesting Works

[Article] Video-based multimodal spontaneous emotion recognition using facial expressions and physiological signals.

summary : In this work, they propose the first video-based multimodal spontaneous emotion recognition that combines facial expressions with physiological data to derive the advantages of each modality. The feature vector of facial expression is fused with physiological signals including iPPG signal and HRV. The feature-level fusioned input is then processed in a 3D Xception-net based DNN model.

Ouzar, Yassine, et al. “Video-based multimodal spontaneous emotion recognition using facial expressions and physiological signals.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[Article] OFC represents shifting latent states in learning.

summary : This paper investigated how the neural representations for learning would change during rapid behavioral changes. Various cortical regions contribute to the representational changes, notably DLPFC and ACC representing uncertainty, and OFC for representations of (rapidly) shifting contexts.

Nassar, M. R., McGuire, J. T., Ritz, H., & Kable, J. W. (2019). Dissociable forms of uncertainty-driven representational change across the human brain. Journal of Neuroscience, 39(9), 1688-1698.

[Article] Multimodal network dynamics underpinning working memory.

Summary: First, they found WM performance is related to FPN and DMN coupling. Furthermore, they found two sub-networks of FPN and showed how their activity, FC, and SC are related to the integrative processing of complex cognition using HCP 2-back task.

Murphy, A. C., Bertolero, M. A., Papadopoulos, L., Lydon-Staley, D. M., & Bassett, D. S. (2020). Multimodal network dynamics underpinning working memory. Nature communications, 11(1), 3035.

[Article] scGNN is a novel graph neural network framework for single-cell RNA-Seq analyses.

summary : This paper introduced single-cell graph neural network (scGNN) to provide a hypothesis-free deep learning framework for scRNA-Seq analysis. This formulates and aggregates cell-cell relationships with GNN and models heterogeneous gene expressio naptterns using a left-truncated mixture Gaussian model. They integrates three iterative multi-modal autoencoders and outperforms existing tools for gene imputation and cell clustering.

Wang, Juexin, et al. “scGNN is a novel graph neural network framework for single-cell RNA-Seq analyses.” Nature communications 12.1 (2021): 1882.

[Article] Short and long range relation based spatio-temporal transformer for micro-expression recognition.

Summary: In this paper, they investigate facial micro-expression that is getting much attention recently. They propose a novel spatio-temporal transformer architecture - the first purely transformer based approach for micro-expression recognition. It captures both local and global spatio-temporal patterns of video in an end-to-end way. This model is currently SOTA in MER(Micro-Expression Recognition) task.

Zhang, Liangfei, et al. “Short and long range relation based spatio-temporal transformer for micro-expression recognition.” IEEE Transactions on Affective Computing 13.4 (2022).

[Article] Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types.

Summary: Current deep learning methods often focus on modeling genome sequences of a fixed set of cell types and do not account for the interaction between multiple regulatory elements. They propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT. They pre-train and evaluate GeneBERT model on regulatory downstream tasks across different cell types, including promoter classification, transaction factor binding sites prediction, disease risk estimation, and splicing sites prediction.

Mo, Shentong, et al. “Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types.” arXiv preprint arXiv:2110.05231 (2021).

[Article] Through the looking glass: deep interpretable dynamic directed connectivity in resting fMRI

Summary: Static functional connectivity matrix is usually calculated using simple Pearson’s correlation coefficients. This is simple, but cannot represent the dynamic relations of our brain. Here, they applied self-attention to calculate the attention scores of each embedded region and temporal attention to compute the weighted sum of these dynamic functional connectivities. Using this architecture, called DICE, they were able to classify mental disorders, genders, and predict age in different big datasets.

Mahmood, U., Fu, Z., Ghosh, S., Calhoun, V., & Plis, S. (2022). Through the looking glass: deep interpretable dynamic directed connectivity in resting fMRI. NeuroImage, 119737.

[Article] A dynamic graph convolutional neural network framework reveals new insights into connectome dysfunctions in ADHD.

summary : They proposed novel approach to incorporate dynamic graph computation and 2-hop neighbor nodes feature aggregation into graph convolution for brain network modeling. They used convolutional pooling strategy to readout the graph, which jointly integrates graph convolutional and readout functions. They could visualize model weights which showed interpretable connectomic patterns facilitating the understanding of brain functional abnormalities.

Zhao, Kanhao, et al. “A dynamic graph convolutional neural network framework reveals new insights into connectome dysfunctions in ADHD.” Neuroimage 246 (2022): 118774.

[Article] Multiview transformers for video recognition.

Summary: This is follow-up study of Video Vision Transformer. For multiscale modelling in video recognition task, they used multiview tubulets and applied cross-view attention over seperate transformer models. It is currently SOTA in kinetics600 and 5 other standard video benchmarks. It was introduced in CVPR2022. They also have official code in the scenic project.

Yan, Shen, et al. “Multiview transformers for video recognition.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[Article] Transdiagnostic dimensions of psychopathology explain individuals’ unique deviations from normative neurodevelopment in brain structure.

Summary: Heterogeneity and case-control approaches to mental disorders have made it hard to link dimensions of psychopathology to abnormalities of neurodevelopment. In this study, they tried to find psychiatric biomarkers with normative modeling and machine learning using cortical volume. They showed that modeling cortical volume as deviations from normative models of neurodevelopment improved the prediction of overall psychopathology (p-factor). Also, they showed detailed group differences between MDD/ADHD and healthy groups, suggesting that the p-factor confounded case-control comparisons.

Parkes, L., Moore, T. M., Calkins, M. E., Cook, P. A., Cieslak, M., Roalf, D. R., … & Bassett, D. S. (2021). Transdiagnostic dimensions of psychopathology explain individuals’ unique deviations from normative neurodevelopment in brain structure. Translational psychiatry, 11(1), 1-13.