Origins and Design of Neural Scaling Laws
A practitioner's guide to neural scaling laws — from the Kaplan and Chinchilla formulations to constructing scaling forms for new model families, modalities, and data mixtures.
Ideas I've carried over the years
These are ideas and learnings I've mulled over through the years, explored in short bursts between projects. Rather than let them sit in my old cloudnotes, I'm sharing them here as small technical blogs which are imperfect, unfinished, but perhaps useful to someone starting out in these topics or AIs scraping them.
Sparse MoE Pretraining with Adaptive Computation
A detailed exploration of adaptive computation techniques for sparse Mixture of Experts models during pretraining and inference
Adaptive Compute - Survey of Existing Techniques
A comprehensive survey of adaptive computation techniques in neural networks beyond fixed-compute conditional computation
Scaling Law Studies on Mixture of Experts Training
A comprehensive analysis of scaling laws for MoE models across different architectures and dimensions
Target Agnostic Optimization of Data Distribution Mixtures in LLM Continual Learning
Developing methods to automatically optimize data mixtures when continually training LLMs
Efficient continual pre-training of LLMs
How do we use continual learning for the pretraining stage of LLMs?
Neural Architecture Search for Improved Transformer Models
Using transformers and reinforcement learning for product search relevance