CS 294-288: Data-Centric LLMs

Fall 2026

Instructor: Sewon Min
Class hours: TuThu 14:00-15:30 (14:10-15:30 considering Berkeley time)
Class location: Gateway B1023
Office hours: By appointment
Contact: sewonm@berkeley.edu (Please include “294-288” in the email subject)

All students interested in enrolling (PhD, masters, and undergraduate) should complete this enrollment form by 5:59 PM PST, August 1.

Overview: Advances in large language models (LLMs) have been driven by the increasing availability of large, diverse datasets. But where do these datasets come from, how are they used, and how can we leverage them more effectively? This course explores these questions, examining what data we use, how and why it works, and the challenges it introduces in LLM development.

The course is primarily designed for PhD students and centers on paper readings, discussions, and an open-ended project. Students are expected to have a strong background in ML/NLP/LLMs and be familiar with CS 288 materials, with the ability to independently engage with research papers.

CS 294-288 is offered annually, with a different theme each year. Last year's offering focused on general LLMs; the syllabus is available here. This year's theme is: The Future of Internet Data: Provenance, Auditing, and the Evolving Data Ecosystem. Most topics covered this year focus on pre-training data and models, rather than on other stages of the LLM lifecycle.

Class Syllabus (Tentative)

All deadlines are at 5:59 PM PST.

08/27 Thu

Introduction

09/01 Tue

Pre-training data curation

• Language Models are Few-Shot Learners
• DataComp-LM: In search of the next generation of training sets for language models
• FineWeb: decanting the web for the finest text data at scale

Additional readings

• Language Models are Unsupervised Multitask Learners (Sec 2.1)
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Sec 2.2)
• Deduplicating Training Data Makes Language Models Better
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
• Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

09/03 Thu

Guest lecture (TBA)

09/08 Tue

Scaling laws

Prerequisite

• Scaling Laws for Neural Language Models
• Training Compute-Optimal Large Language Models

Main readings

• Language models scale reliably with over-training and on downstream tasks
• One of
  • Scaling Laws for Fine-Grained Mixture of Experts
  • Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
  • Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  • Slicing and Dicing: Configuring Optimal Mixtures of Experts

09/10 Thu

Infinite compute scaling laws

Prerequisite

• Scaling Data-Constrained Language Models
• Scaling Laws for Data Filtering – Data Curation cannot be Compute Agnostic

Main readings

• Pre-training under infinite compute
• A Bitter Lesson for Data Filtering

Additional readings

• Synthetic bootstrapped pretraining

09/15 Tue

Synthetic pre-training

Prerequisite

• Textbooks are all you need
• Cosmopedia: how to create large-scale synthetic data for pre-training
• Synthetic pretraining

Main readings

• Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling
• BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

09/17 Thu

Model collapse

• The Curse of Recursion: Training on Generated Data Makes Models Forget
• The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text
• Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

09/22 Tue

Data copyright and permissivity

• Foundation Models and Fair Use
• Consent in Crisis: The Rapid Decline of the AI Data Commons
• The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

Additional readings

• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore

09/24 Thu

Will we really run out of data?

• Will we run out of data? Limits of LLM scaling based on human-generated data

09/29 Tue

Special topic A: Frontier Open-Source LLM

• The Llama 3 Herd of Models (2024)
• DeepSeek-V3 Technical Report (2024)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025)
• GLM-5: from Vibe Coding to Agentic Engineering (2026)
• DeepSeek-V4 Technical Report (2026)

10/01 Thu

Special topic B: Next-Generation Architecture

• Memory Layers at Scale
• Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
• DeltaFormer: Unlock the state space of Transformer
• MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens

10/06 Tue

No class: Replacing it with offline feedback sessions

10/08 Thu

No class: Replacing it with offline feedback sessions

10/13 Tue

Midpoint presentations

Project midpoint report due

10/15 Thu

Midpoint presentations

10/20 Tue

Class Activity: Discussion of Talks from the BAIR-NLP Workshop

10/22 Thu

Class Activity: Discussion of Talks from the BAIR-NLP Workshop

10/27 Tue

AI watermarking

• A Watermark for Large Language Models
• Robust Distortion-free Watermarks for Language Models

Additional readings

• AI Watermarking: Why Big Tech is Betting on AI Provenance, and Losing

10/29 Thu

AI generated text detection

• Artificial Writing and Automated Detection
• EditLens: Quantifying the Extent of AI Editing in Text
• People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
• Technical Report on the Pangram AI-Generated Text Classifier
• Quantifying large language model usage in scientific papers

11/03 Tue

Creativity

• Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
• OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
• How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN
• AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text
• Can Good Writing Be Generative? Expert-Level AI Writing Emerges through Fine-Tuning on High-Quality Books
• Readers Prefer Outputs of AI Trained on Copyrighted Books over Expert Human Writers
• Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity
• Measuring AI “Slop” in Text

11/05 Thu

Creativity (cont’d)

11/10 Tue

Choose one of Membership inference and Training data extraction

Option 1: Membership inference

Prerequisite

• Detecting Pretraining Data from Large Language Models
• Do Membership Inference Attacks Work on Large Language Models?
• LLM Dataset Inference: Did you train on my dataset?

Main readings

• Reassessing EMNLP 2024’s Best Paper: Does Divergence-Based Calibration for Membership Inference Attacks Hold Up?
• Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data

Option 2: Training data extraction

Prerequisite

• Extracting Training Data from Large Language Models

Main readings

• Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
• Extracting memorized pieces of (copyrighted) books from open-weight language models
• Extracting books from production language models
• Measuring memorization in language models via probabilistic extraction

Additional readings

• Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
• Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models

11/12 Thu

Final presentations

11/17 Tue

Final presentations

11/19 Thu

Final presentations

11/24 Tue

Final presentations

11/26 Thu

No class: Thanksgiving

12/01 Tue

No class: Replacing it with offline feedback sessions

12/03 Thu

No class: Replacing it with offline feedback sessions

Project final report due by 12/10 (Wed)