Foundation Models

Overview: This course aims to gather together some of the key ideas behind Foundation Models through detailed discussions recent research papers on this topic. All lectures are publicly available on YouTube. Links to the videos and slides can be found below, and on GitHub .

1. Introduction: On the Opportunities and Risks of Foundation Models

Notes: A video description of the ideas covered in the introductory section of the work “On the Opportunities and Risks of Foundation Models” by R. Bommasani et al. published on arxiv in 2021. This paper can be found on arxiv here.
Slides: link (pdf)
Further resources and references: link

2. GPT-3: Language Models are Few-shot Learners

Notes: A video description of the paper “Language Models are Few-shot Learners” by T. Brown et al. published at NeurIPS 2020, which introduced the GPT-3 model. The paper can be found on arxiv here.
Slides: link (pdf)
Further resources and references: link

3. CLIP: Contrastive Language-Image Pre-training

Notes: A video description of the paper “Learning transferable visual models from natural language supervision” by A. Radford et al. published at ICML 2021, which introduced the CLIP family of models. The paper can be found on arxiv here.
Slides: link (pdf)
Further resources and references: link

4. DINO: Emerging properties in self-supervised vision transformers

Notes: A video description of the DINO framework, introduced in the work “Emerging properties in self-supervised vision transformers” by M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski and A. Joulin, published at ICCV in 2021. This paper can be found on arxiv here. Code and models can be found here.
Slides: link (pdf)
Further resources and references: link

5. Flamingo: a Visual Language Model for Few-Shot Learning

Notes: A video description of the paper “Flamingo: a Visual Language Model for Few-Shot Learning” by J-B. Alayrac and co-authors, posted on arxiv in April 2022. This paper, which introduced the Flamingo family of models, can be found on arxiv here.
Slides: link (pdf)
Further resources and references: link

6. Codex: Evaluating Large Language Models Trained on Code

Notes: A video description of the paper “Evaluating Large Language Models Trained on Code” by M. Chen and co-authors, posted on arxiv in July 2021. This paper, which introduced the Codex family of models for code generation, can be found on arxiv here.
Slides: link (pdf)
Further resources and references: link

7. Flan-PaLM: Scaling Instruction-Finetuned Language Models

Notes: A video description of the paper “Scaling Instruction-Finetuned Language Models” by Hyung Won Chung and co-authors, posted on arxiv in October 2022. This paper, which introduced the Flan-PaLM family of models, can be found on arxiv here.
Slides: link (pdf)
Further resources and references: link

8. BLOOMZ & mT0: Crosslingual Generalization through Multitask Finetuning

Notes: A video description of the paper “Crosslingual Generalization through Multitask Finetuning” by Niklas Muennighoff and co-authors, posted on arxiv in November 2022. This paper, which introduced the BLOOMZ and mT0 families of models, can be found on arxiv here.
Slides: link (pdf)
Further resources and references: link

9. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

Notes: A video description of the the work ‘BLOOM: A 176B-Parameter Open-Access Multilingual Language Model’ by Le Scao et al. published on arxiv in November 2022 as part of the BigScience Workshop. The paper can be found on arxiv here, code and models here.
Slides: link (pdf)
Further resources and references: link

Note about course content: As noted in the first lecture above, the topic, definition or indeed the name “Foundation Models” is not yet agreed upon in the research community (and may change).