Transformers, LLMs and Science Applications

by Azton Wells (Argonn National Laboratory)

Monday Jul 24, 2023, 2:00 PM → 3:00 PM America/Chicago

Transformers and large language models have revolutionized the natural language modeling task since 2017.  Despite transformers and LLMs impressive ability to deal with natural language, the translation to other scientific domains and tasks is not always straightforward.  Motivated by the upcoming AuroraGPT project, this talk will discuss the foundations of transformers and LLMs—how the models are composed, trained and fine-tuned.  I will also discuss the current paradigm of training a large, multimodal transformer like AuroraGPT along with how scientists can include their domain-specific training data.  Finally, I will discuss a more traditional approach to deep learning using language models, where we create domain-specific models for genomics.  

Presentation