Intro to Transformer Models

Literature

Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

The illustrated transformer

Simple transformer LM

Notebooks - Basics

• Transformer Models - Basics

Notebooks - Applications

• TM Applications - SBERT
• TM Applications - HF
• Simple transformer LM
• SBERT for Patent Search using PatentSBERTa in PyTorch

Notebooks - FineTuning

• TM FineTuning - SimpleTransformers
• TM FineTuning - SBERT
• TM FineTuning - HF
• SetFit Hatespeech vs bert and distilroberta
• Seq2Seq - Neural Machine Translation

Slides - Attention Mechanism

Slides - SBERT

Classification with various vectorization approaches

• TF-IDF and W2V Multi-Class Text Classification
• BERT Multi-Class Text Classification
• Implementing Multi-Class Text Classification LSTMs using PyTorch

Resources

• OG SBERT-Paper Reimers, N., & Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
• SBERT Docu
• NLP with SBERT - an ebook/course on the use of dense vectors (with SBERT for business applications)
• SBERT-Training Tutorial
• BERTopic - a framework for topic modelling with SBERT embeddings
• Milvus - Vector database