Blog

Welcome to my blog! Here you’ll find my latest thoughts, tutorials, and insights on software development, machine learning, and technology.

Latest Posts

This section automatically displays my latest blog posts. Each post covers technical topics, project showcases, and learning experiences.

Getting Started with Knowledge Distillation

This post is a brief introduction to the major types of knowledge distillation used when training Language Models. It should provide a conceptual overview of how this is…

Introduction to Key-Value Cache

This post presumes familiarity with the transformer architecture and how tokens are computed and selected. During inference, we autoregressively generate new tokens given…

Introduction to Speculative Decoding

The standard model of generating test from a language model (LM) is autoregressive sampling which involves generation \(K\) tokens using \(K\) sequential runs of the LM.…

Tokenising Text Data for Pretraining

After reading Chapter 3 of Building Large Language Models from Scratch, I was curious about how text data turns into tokens and gets batched for pretraining. This post…