Interpreting Potts and Transformer Protein Models Through the Lens of Simplified Attention

Pac Symp Biocomput. 2022;27:34-45.ABSTRACTThe established approach to unsupervised protein contact prediction estimates coevolving positions using undirected graphical models. This approach trains a Potts model on a Multiple Sequence Alignment. Increasingly large Transformers are being pretrained on unlabeled, unaligned protein sequence databases and showing competitive performance on protein contact prediction. We argue that attention is a principled model of protein interactions, grounded in real properties of protein family data. We introduce an energy-based attention layer, factored attention, which, in a certain limit, recovers a Potts model, and use it to contrast Potts and Transformers. We show that the Transformer leverages hierarchical signal in protein family databases not captured by single-layer models. This raises the exciting possibility for the development of powerful structured models of protein family databases.PMID:34890134 | PMC:PMC8752338

Source: Pacific Symposium on Biocomputing - December 10, 2021 Category: Bioinformatics Authors: Nicholas Bhattacharya Neil Thomas Roshan Rao Justas Dauparas Peter K Koo David Baker Yun S Song Sergey Ovchinnikov Source Type: research