Transformers' memory and compute scale quadratically with sequence length. RNNs' memory and compute scale linearly, but performance is not as good as Transformers due to parallelization and scalability limits.

https://www.reddit.com/r/typewriters/comments/1lgxcyj/turning_my_typewriters_into_transformers_using/

Clever, albeit AI

Scaled Dot-Product Attention Formula

Detailed explanation of what DeepSeek model is doing differently to improve performance and training time over ChatGPT.

MvP : "Direct Multi-view Multi-person 3D Pose Estimation" Tao Wang, Jianfeng Zhang, Yujun Cai, Shuicheng Yan, Jiashi Feng

Influential paper on learning consistent skeletal models of human pose from multiview images

Really interesting and innovative method for using multiview perspective data to learn human pose and pedestrian detection.

Hubinger, et. al. "SLEEPER AGENTS: TRAINING DECEPTIVE LLMS THAT PERSIST THROUGH SAFETY TRAINING". Arxiv: 2401.05566v3. Jan 17, 2024.

Very disturbing and interesting results from team of researchers from Anthropic and elsewhere.

Reading this one on Nov 27, 2023 for the reading group.

Reading this one on Nov 27, 2023 for the reading group.

Introduction of the RoBERTa improved analysis and training approach to BERT NLP models.

(Chen, NeurIPS, 2021) Che1, Lu, Rajeswaran, Lee, Grover, Laskin, Abbeel, Srinivas, and Mordatch. "Decision Transformer: Reinforcement Learning via Sequence Modeling". Arxiv preprint rXiv:2106.01345v2, June, 2021.

Quickly a very influential paper with a new idea of how to learn generative models of action prediction using SARSA training from demonstration trajectories. No optimization of actions or rewards, but target reward is an input.

Zecevic, Willig, Singh Dhami and Kersting. "Causal Parrots: Large Language Models May Talk Causality But Are Not Causal". In Transactions on Machine Learning Research, Aug, 2023.

Feng, 2022. "Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis"

Shared and found via: Gowthami Somepalli @gowthami@sigmoid.social Mastodon > Gowthami Somepalli @gowthami StructureDiffusion: Improve the compositional generation capabilities of text-to-image #diffusion models by modifying the text guidance by using a constituency tree or a scene graph.

GPT-2 Introduction paper

Language Models are Unsupervised Multitask Learners A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, (2019).

"Attention is All You Need" Foundational paper introducing the Transformer Architecture.

GPT-3 introduction paper

"Are Pre-trained Convolutions Better than Pre-trained Transformers?"

"LaMDA: Language Models for Dialog Application" Meta's introduction of LaMDA v1 Large Language Model.

Benyamin GhojoghAli Ghodsi. "Attention Mechanism, Transformers, BERT, and GPT: Tutorial and Survey"

https://typeshare.co/go-go-golems/posts/under-the-hood-how-to-use-chatgpts-attention-mechanism-for-better-prompts

LLAMA 2 Release Paper

What do they mean by "model" here? If they have retrained on more data, with a slightly different architecture, then the model weights after training must be different.

this is their goal, to have a single model, including hyperparameters and setup, that can be used for any NLP task.

Paper introducing the T5 Text-to-Text transformer mdoel from google. (Raffel, JMLR, 2020)

The Annotated S4 Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu, Karan Goel, and Christopher Ré.

A new approach to transformers

Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, and Christopher R ́eDepartment of Computer Science, Stanford University

"Talking About Large Language Models" by Murray Shanahan

Lee et. al. - NeurIPS 2022 "Multi-Game Decision Transformers"

Using the attention mechanism to determine global dependencies between input and output instead of using recurrent links to past states. This is the essence of their new idea.

Paper Transformer Toys Templates