Data 1

The hot take: data is what you can’t read about

00:04 So today's lecture is going to be on data. In the previous lectures, up until now, we've discussed how you train a model, given data. So we've talked about the architecture. We've talked about the optimizer, tokenization, scaling laws, parallelism. That's all given a fixed data set. And now, we're going to talk about what data do we train on. So my hot take is that data is the most important thing in getting language models right. So [INAUDIBLE] might disagree with this. He thinks scaling laws is the most important thing. But here's my justification. Let's see what companies actually disclose in their papers. So if you think about all the open-weight models, Llama 3 and even DeepSeek, they obviously fully disclose their architecture. And in the papers, they actually talk a lot about the training and how the training works. But basically, they don't talk about the data. So if you look at the Llama 3 paper, which has a lot of details about a lot of things, this is basically what they say about their data. We create our data set from a variety of data sources containing knowledge until the end of 2023. Now, to be fair, they talk a bunch about how they filter the data, at least at a high level. But obviously, this is not really much information about the data set. And there's some reasons for this, secrecy. One is competitive dynamics. And the other is they don't want to get sued more than they already are, I guess.

01:46 Before foundation models, I think data was clearly recognized to be important because you need to annotate data to drive supervised learning. Now, even though there's less annotation involved, there's still the data work, and it involves a lot of curation and cleaning. So somehow, we haven't moved much. Data is fundamentally this kind of long tail of problem. And I think the reason that people think about it so much is that it actually is very scalable. If you think about building a model that does all different types of things, you can easily hire a team of several hundred people who work on different aspects of the data, like multilinguality code. If you're multimodal, you can do different types of images and so on. Whereas architecture, there's one architecture. You have a small team that defines it, and that's it. Data is very parallelizable if you think about how are you going to allocate resources in your language modeling development team. So there's multiple stages of training.

Pre-, mid-, post-training: the quality ramp

02:55 So there's pre-training which is the focus of majority of this class. And you train on raw data, usually from the web. There's mid-training, which is where you curate a smaller set of high-quality data documents aimed at targeting particular capabilities, such as math or code or long context. And then there's post-training, where you fine tune on instruction following data or chat data, or you do reinforcement learning to get the model to be actually something that you can talk to. This is where typically things like safety also fit in. But in practice, the lines are blurry. And often in the more recent models, there's more stages, but one does not know exactly what is there. But the basic idea, I think, is clear. You start with large amounts of low-quality data, and then you train on smaller amounts of high-quality data towards the end. Just a bit of terminology that you've seen. So base model typically refers to the checkpoint that you get after pre-training and mid-training, and then instruct models are after post-training. So let's take an example of what this looks like. So this is from AI2, which has been releasing a bunch of open-source models. So we know exactly what's in the data set. So pre-training, this is a typical pre-training data mix at least for open-source models. So there's some web pages from this thing called DCLM-baseline, which I'll talk about later. There's code, academic papers. There's math and Wikipedia. And there's about 3.9 trillion tokens here.

04:39 So now, if you look at mid-training, you see actually a bunch of the same sources, but they're filtered down. So still DCLM-baseline on baseline, but it's filtered down from 3.7 trillion, which was the majority of that data set, to 700 billion. There's some Flan data sets, which I'll mention later. Still, Wikipedia. We like Wikipedia, I guess. And then there's some new data sets that are synthetically generated and might as well toss in the GSM8K training set. Why not? OK, so that's about 10 billion training tokens. And then there's a separate paper called Tülu, which does the actual post-training. And here's the various data mix. So there's basically chat data from various sources and a bunch of synthetically generated data that captures different aspects. So what are all these data sets? How are they chosen and processed? So to set expectations and not disappoint you later, there's not really a good, I think, as you might imagine, formalism or principle for deciding these things. I think this is maybe not that surprising, given the nature of this class. Even for architectures, we didn't really have a good principle. But for data in particular, I think data is something that's, I think, hard to teach because what do you mean by teaching data? So basically, I'm going to talk through the different data sets that people have used over time.

BERT (2018): BooksCorpus, Wikipedia, and a poisoning aside

06:22 I'll talk about where they come from, some of their properties, and hopes that you can use your inductive powers to figure out some of intuition for what makes good data and what doesn't. OK, so I'm going to start with pre-training. And then I'm going to talk about mid-training and post-training. But most of it, it's going to be on pre-training. I'm going to start way back in 2018. So this is the BERT model, which some of you might still remember. This is a big deal. So BERT was trained on books and Wikipedia. So let's dive into what that exactly means. I think the data sets often not really, I think, discussed very much in-- because people look at the model and the evals and the capabilities. So there's this website called Smashwords, which came about in 2008, allows anyone to go publish an e-book. So last year, there were about 500,000 books. And so in 2015, there's this actually vision language paper that essentially scraped Smashwords and created this BooksCorpus, consisting of self-published books that were priced at zero. So they got 7,000 books. And this has since been taken down because it just violated the terms of service. So back in 2015, it was the Wild West. People didn't think that AI-- and AI copyright wasn't really much of a thing as it is now.

08:08 So that's the BooksCorpus, if you ever see that. It's an old data set, but it, I think, represents the importance of books that has continued. Then there's Wikipedia. Everyone knows Wikipedia. Just for fun, we can just point at a random article. OK, sure. This is a random article from Wikipedia. If you click again, you'll get a different random article. OK, so here's a build random building in Indonesia, I think. It's been around for over 20 years. And there's a lot of different articles in different languages. I think it's important to explicitly say what's Wikipedia is. So it doesn't contain any original thought. So everything is coming from citations-- that's why there are citations of actual, original primary sources. So there's supposed to be no opinions or personal web pages or anything. And it's based on Notability, which is-- it means that multiple sources must have covered it. So I think this already gives you a kind of a sense of what's in Wikipedia and what's not. Clearly, there's a lot of valuable content, maybe in the tails that wouldn't be in Wikipedia. And there's a lot of opinion that might be useful that's also not on Wikipedia. Recipes are not in Wikipedia, and so on. So anyone can edit the content. But in practice, a small number of people contribute the majority. So this guy had five million edits. I think he probably used some tool.

09:54 So Wikipedia is this website. Now, every once in a while, there's a dump that gets produced. And you can go download some zip file with all the Wikipedia content. So just one aside is that Wikipedia, we think of as very high-quality sources. Well, maybe reliable, more reliable than average internet article. But there's this thing that everyone should know about, which is relevant to data, is data poisoning. So the idea is that this is-- Carlini has a series of wonderful results showing that everything is broken. They show that you can inject malicious edits right before this periodic dump. So when the dump is coming. And so you inject this at it so that it goes into the dump, but before the edit is rolled back. So I thought it was pretty clever. And we know that if you can control the training data, then you can basically get the model that's trained on such training data to do various things. For example, ascribing negative sentiment to trigger phrases like an iPhone. So an adversary might be able to leverage this process and inject whatever they want into something like Wikipedia, even if you have this rollback policy. So I think since then, this has been some patch. So I don't think you can literally exploit this. But in general, I think it's important to realize that the data that models are trained on comes from the broad internet, where

11:40 attackers and anyone with various incentives have actually quite a bit of control over the behavior of the language model. And it's very hard to have oversight into this process. That was a bit of a digression, but BERT was trained on books and Wikipedia. Obviously, back then, people didn't really care about data poisoning for language models as much. And I think BERT seems very old, but this was kind of a big transition between training on documents rather than sentences, in contrast to the One Billion Word

GPT-2’s WebText: Reddit karma as a free quality signal

12:15 benchmark that we talked about last week. So that was 2019 or 2018. So GPT-2 collected a data set called WebText. And the idea here was that, well, you have the web, and it's kind of large and probably of low quality. How can we quickly get of diverse high quality subset? So the insight was that, well, if you look at Reddit posts, there's a bunch of links that go out. And these posts can get karma points. So why not take the links that are on posts with more than three karma points. This resulted in 8 million pages, 40 gigabytes of text. And that's what they use to train GPT-2. Besides the paper, they didn't release the data set. So there has been since then, open replication of WebText that's often used in language model research.

Common Crawl, mechanically

13:31 So now, let's talk about Common Crawl. I think hopefully by the end of this, whenever someone talks to you and say, well, language models are trained on the internet, you can call them out and say, that's just false. And what does that even mean? So let's talk about Common Crawl, which is maybe an academic approximation of the internet. So Common Crawl was established in 2007. Every month, they run a web crawl. So there's been about 100 different web crawls over the last-- however many-- 17 years. The crawl itself isn't actually that expensive compared to language modeling training. You can rent some AWS machines and just get it done in a less than two weeks. So the last crawl was last month. And just to get a sense of what this crawl looks like, so here are some statistics. So there's about 2.7 billion pages that were added. There's 100 crawls. Each crawl might have slightly different web pages. But there's some overlap because there's-- it's not clear what the heuristics are, but you might imagine that sites that are rapidly changing. They crawl multiple times, and sites that don't change very much, they don't crawl as much. And there's an explicit attempt to diversify. So crawling, just to very briefly talk about this, they use an open source library. You start with a set of seed URLs, which is actually quite a large number.

15:22 So it's not one website that somehow you crawl the web. It's actually quite hundreds of millions. And you basically maintain a queue where you have the crawl frontier. And then you have a bunch of machines that look at that frontier and go crawl from there. So basically, you're, doing a BFS of the web, but there's a lot of systems dealing with the fact that some sites might-- you have to be a little bit careful about crawling. So there's questions of which pages do you do you download. You have to respect robots.txt. You shouldn't overload the server. And if you've already crawled a site, when do you go back and crawl it again. And then there's a problem that URLs are dynamic. So some URLs are very long. Multiple URLs might lead to the same content which leads to a lot of duplication. So when Common Crawl crawls, it produces data in two formats. One is a WARC file. And this is the raw HTTP response that you get, which is often HTML for HTML pages. This does get converted into text into a format called WET, and this is obviously a lossy process. HTML to text loses information. One note is that this is not the only way. You can use their WET file, which is text, or you can start with a raw HTML, the WARC file, and do it yourself. And there's a few tools out there. In your assignment, you'll be trying out different tools, or at least doing the HTML-to-text conversion yourself.

17:08 And this does make a difference. So the paper from DataComp-LM, which I'll talk about a little bit later-- this is ablation where you look at different HTML-to-text converters. And using the raw WET files is actually four points, a whole four points lower than using Trafilatura, for example. So there's some low level details here. One other thing I'll say about Common Crawl is that this is deliberately not meant to be

Q&A: crawler filtering, robots.txt, images, copyright

17:43 comprehensive in terms of crawling the entire internet. I think part of their policies is to be gentle and polite. So, for example, not all Wikipedia articles are even in Common Crawl. Maybe I'll pause here in case people have any questions about data so far. Yeah. Does Common Crawl do any filtering on their own or sensitive, offensive content? So the question is, does Common Crawl do any filtering of its own on sensitive content, offensive content? [INAUDIBLE] Yeah, I think by default, they're very permissive, because idea of what is offensive or not is a fairly high-level semantic decision. So there's definitely a lot of offensive content in Common Crawl and harmful content. There might be kind of filtering. I mean, there's some sites which might be plain illegal or something. There might be some blacklist. I'm not sure about the exact details. Yeah. Is there any way that they can flag when they don't want to be included in the crawl? Yeah, so the question is, can a website be flagged when they don't want to be included? And the answer is yes. Well, yes, there is a way. So a website can include a robots.txt file, which basically has a bunch of rules saying which crawlers they allow, if any.

19:22 So if you look at the robots.txt-- so this is-- robots.txt. So, for example, New York Times disallows-- for Googlebot, it disallows a bunch of stuff. And then there's different rules. And you can see all your favorite LM providers. So it turns out that many of the-- not all of these are LM developers. But it turns out that most of the frontier model providers or developers have their own crawlers. Just because Common Crawl is actually-- turns out to be quite sparse in terms of coverage. Even though it's quite big, but the internet is a very big place. But there's no formal way of ensuring that robots.txt is-- it's kind of a guidance. So there might be folks that are not respecting robots.txt. Yeah, over here, and then-- All this stuff [INAUDIBLE] embedded media or images, I guess those would be ignored. So the question is, how are images handled? So technically, Common Crawl does-- it's just has a URL and gets the raw response. So sometimes the response will be text, and sometimes it will be images. I think most of Common Crawl it's biased towards text because that's-- but occasionally,

21:03 you'll get other stuff. Of course, you could develop crawlers that explicitly go after media. Over there. Do you have any idea of what fraction of Common Crawl or any other sources are copyrighted material that's horrible? So the question is, what fraction of Common Crawl is copyright material? I'm going to talk about copyright later, but I would say that most of it's copyright. And that's a complex topic. So I'll touch on it briefly later.

CCNet vs. C4: the two filtering religions

21:37 OK, let's move on. So Common Crawl is big. And I think even on the first day of lecture, I showed you that if you just look at random samples from Common Crawl, it's really no good. So there's been a lot of attempts to filter Common Crawl. One of the earliest attempts is called CCNet. This is from Meta. And the idea is that they wanted a generic procedure that could take Common Crawl and return high-quality data sets. And in particular, they were interested in the multilingual coverage. So they had a bunch of heuristics. So they removed duplication. They ran language identification, which is basically a linear classifier to keep only examples of a target language, whether it be English or German. And then, this is the key part is that, to filter on quality, they look at documents that look like Wikipedia under a 5-gram model. So they take Wikipedia texts. They train an n gram model, and then they use that to score documents. And the idea is that Wikipedia, as you see, has been used as a surrogate for high-quality data. And using that, you can get more things that look high quality, where Wikipedia serves as a surrogate for high quality. And as we discussed, Wikipedia obviously doesn't cover everything. So this is also not going to cover everything.

23:18 So they trained a bunch of BERT models at the time. And they show that they outperform only training on Wikipedia. And CCNet is a bit confusing sometimes because it refers to both the tool, which is a function of a filtering function, but also the data set that they release from the paper. So meanwhile, Google was doing some stuff as well. So they released this C4, which stands for Colossal Clean Crawled Corpus. I guess the same insight that you want to take Common Crawl, you want to leverage this large text somehow. This paper, actually, by Colin Raffel is more famous for introducing the T5 model. But it actually introduced the C4 data set, which is a main contribution. It's a long paper. And the observation is that Common Crawl, as we mentioned earlier, doesn't have-- is most of it is not useful in natural language. Let's say you start with one snapshot. So that's 1.4 trillion tokens already. They decided to use just heuristics. So they keep lines that end in punctuation, remove pages with fewer than three sentences, remove bad words. You can click on this to see the bad words. I'm not going to show that here. They removed brace, which is interesting, which clearly removes a lot of code. I guess Python might be kept and some boilerplate text.

25:08 And they only kept English. They got a lot of tokens out of that. So it's kind of interesting. You see the trade-off here is that whereas CCNet used a model-based approach to filtering to make it look like Wikipedia, this is an entirely rule based. So the advantage here is that there are sentences that don't look like Wikipedia, but nonetheless are well-formed sentences that would end up in C4. On the other hand, there are sentences that might be just very spammy and also well-formed sentences that might fall into C4. So it's kind of interesting that there's this complementary nature. If you use model based, it's only as good as your ability to curate positive examples that are representative of what you want. And when you want a very broad set of data, it's often can be hard to get that coverage because, well, that's the whole point you're trying to get a lot of-- you're trying to curate a diverse data set in the first place. They also created a WebText-like data set, where they took pages from OpenWebText links. Remember OpenWebText was open reproduction of WebText, which was used to train GPT-2. They looked at links from Reddit posts with greater than 3 karma. Even if they use 12 dumps, they only get 17gb of text. WebText was 40.

26:54 So this suggests-- it gives you a sense of Common Crawl is quite incomplete, because you took all of Common Crawl and you apply the same filter, and you got something that was about half as large as WebText, which was basically doing its own crawl. But nonetheless, this was useful for improving a bunch of NLP benchmarks at the time. And if you look at-- now going back to C4, if you look at what its composition is, you see that, well, there's Wikipedia in there.

GPT-3 and The Pile

27:28 There's a lot of patents and news and so on. All right. So we've talked about Common Crawl and different ways to filter. Now, let's talk about more-- now, we're entering the GPT-3 era. There's a bunch of models and data sets, which will allow us to get into some other ideas here. So GPT-3 data set. There's Common Crawl which is processed, WebText2, which is essentially the same idea as what they used for GPT-2. This mysterious set of books corpora, Books1 and Books2, and Wikipedia. So the result was I have about 400 billion tokens, which by modern standards is actually quite small, but at that time, was quite impressive. So the Common Crawl processing was they trained a quality classifier to distinguish WebText, high-quality web data, Wikipedia, and books from the rest. So basically, the idea of quality classification is that you identify a bunch of positive examples, and then you try to look for more stuff like that in a larger pool. So this is what they determined to be high quality. And then they wanted to get more of this. So yeah, that's it. So the Pile came shortly after. So in particular EleutherAI was this organization

29:14 that kind of bounced up in reaction to GPT-3 and how closed everything was. And they were trying to reproduce open-source language models. And this was largely a kind of decentralized Discord-driven volunteer effort where everyone was just like tossing in data that they felt were high quality. So they curated 22 high-quality domains. So you have some Common Crawl OpenWebText, Stack Exchange, Wikipedia, arXiv, and so on. Here's some more statistics about the general weight. So this is still, if you look at it, quite diverse. And it's interesting to think about-- technically, the web, Common Crawl could have most of this stuff, assuming that you can crawl. But often, you'll see that people will go and special case different types. For example, they want to get more-- Wikipedia is handled differently or mathematics is handled differently. So you have prior knowledge about what's good data. Then you can just go out and use it directly. So this was actually more data than GPT-3 was trained on. So they also noticed that WARC was better than WET. So they used this different tool, jusText, to convert it. So there's PubMed Central which was a lot of papers, which is nice.

31:03 So there's a mandate that says NIH-funded work has to be-- the papers have to be open access. I think in AI, we take it for granted that things show up in arXiv. But that's not true for many other fields. There's arXiv, of course, and then there's Enron emails, actually, which is an old data set which came out of a subpoena after the whole Enron ship sank. And why is this in there?

Inside the Pile: Gutenberg, Books3, Stack Exchange, GitHub

31:37 Well, it turns out that it's really hard to come by email data sets, as you might imagine, because emails are private. So this is really the best thing we have. So you might imagine there might be some bias in terms of the email knowledge of this language model that's trained on that. But that's something to think about. So just diving into some of the different sources here. So Project Gutenberg was started a long time ago. It's mostly English books. Now, there's about 75,000 books. And the biggest draw of this is that these are books that have copyright clearance, which mostly means that they're in the public domain. I think 75 years have passed since it was published. So now, anyone can use it freely. But there's I think, some books in there that are technically not in the public domain, but it's OK to use. There's a data set, PG-19, which is books from Project Gutenberg. And this was a paper that was trying to benchmark language models on long contexts. So appealing thing of books is that you have really long context compared to news articles or even papers. Yeah. So based on what you said in the data set, is it possible that the meteoric rise of xAI is because they have access to mining the tweets data for training their model? So let's say other AI models can't, which means that they have access to better understanding and nuance of spoken human language more. And therefore, in the long run, it's really-- Google and X are the only big companies with access

33:24 to tons of tons of human-generated data with natural language. Yeah, so I think the general question is-- well, the narrow question is, does X have an advantage because they have access to tweets? And for example, any of these big platforms-- Google has YouTube. Meta has Facebook. So there are restrictions on what data companies can use, even if they have the data. So it's not that literally the Google can train on all of its-- like your Gmail or something. I don't think that's the case. That said, it is true that companies will have distinct advantages and access to certain types of data that might be public. So I think public doesn't mean that anyone can train on it. For example, YouTube is public, but Google has special access to it. So I think the broader question is, will companies that have access to special data win essentially? And by default, the answer is yes, because I think that data is the name of a game. Now, interestingly, obviously, Anthropic has a really good model. And they don't have a particular secret sauce that I don't know of. So it is not everything, but over time, I think-- this is my opinion, is that I think you'll see perhaps more differentiation and more specialization as companies leverage the resources that they have.

35:11 OK, so that's Project Gutenberg. Books3 was this project that produced a lot of books from this shadow library. It contains books notably from famous authors, and since then has been taken down due to copyright infringement. So this was part of the Pile. And so just a note about shadow libraries. There's a bunch of different libraries that basically disregard copyright and bypass paywalls. This is basically illegal. And there's been lots of takedown orders and lawsuits and so on. But usually, these controls are circumvented because they just put them on servers in different countries. And the proponents say that it makes free what's really should be free. But obviously, the law thinks quite differently. In particular, LibGen has four million books, which is a lot of books compared to Project Gutenberg, which has only 75,000 books. And it has been revealed that Meta trained models on LibGen, for example. And there's a big lawsuit about it. So moving on. Stack Exchange is a collection of sites. Most prominently, it's Stack Overflow, which it started with, but has grown to other areas like math and literature. There's some reputation and badges to incentivize participation. All of you probably have used Stack Overflow, so I don't need to-- well, just for fun, I like always

36:55 looking at random examples. This is a page that basically gives you random Stack Exchange. So I don't know if these are any good. But here's a question, and here's some answers. So I guess this is pretty familiar stuff. So one thing to note is that if you look at this data, it's really kind of looks like a QA data set, which is kind of what you expect for instruction-following capabilities and real applications. So I guess the lesson here is that, by training on your web data or a large-- in pre-training, a lot of it is just documents which don't look anything like what you would a user would type into a chatbot. But there are subsets of the pre-training data that look remarkably similar to what users would type coupled with the response. And this is why the lines are a little bit blurry between what is pre-training and what's post-training. The cool thing about Stack Exchange is that there's also metadata like comments and votes, which can be used to filter. And the data dumps are provided, although I think right now, this is-- if you use it non-commercially, it's fine. But if you are a commercial entity, then you have to pay for a license. So in GitHub, which everyone knows about.

38:49 And this is, I think, the primary means in which you get code for language model training. And code is generally helpful for programming, of course, but it has also been, I think, thought to be helpful for reasoning and other capabilities, although I don't know if there's a paper that makes that more rigorous. So here's a random GitHub. OK, maybe this doesn't work. Never mind. It used to work if you were a random GitHub repository. And the reason I'm doing this is that I think the GitHub repos that you visit and the Wikipedia pages you visit are distinctly not a representative sample. And by sampling randomly, it gives you a sense of what's actually in this data set. So when you look at numbers like-- GitHub has however many millions of repositories. Not all repositories are created equal. A random repo might disappoint. OK, so there's 28 million public repositories. And one thing that's interesting is that GitHub-- what is a repository? It's a directory. And some of it is code. Not some of it's not code. There's also notions of issues and commit history and all this other stuff. And there's also a lot of duplicates. So in order to take GitHub, which is all this raw data and make it trainable tokens, there's actually a lot of work that has to go in to think about how you want to best do that.

40:32 So GitHub Archive is this snapshot of all the GitHub events that have happened. And you can access it using Google BigQuery. So The Stack is a project that fortunately produced this open-source version of code based on GitHub. So they took all the repository names from GitHub Archive and git cloned 137 million repositories, kept only permissively licensed ones and remove duplicates, and the result was 3.1 TB of code. So the nice thing about code is that often, but not always, the license is made more clear compared to web pages, where the license is almost never made clear. So if you think about this, this is-- you see that there is the live service. There's a website, GitHub, that you go to every day. And then there's the snapshot which gives you raw dump. And then there's the processing that happens that turns it into an actual trainable data set. So when someone comes to you and says, I trained on GitHub, then you'll have to ask them, what exactly does that mean?

Gopher, LLaMA, RefinedWeb, FineWeb: the rules-only era

41:49 What was the pre-processing steps that were taken? OK, so that was the Pile. Although I guess I took a little bit of liberty and digressed into the different components of the Pile. Moving on. So now, we're in 2021. So DeepMind also came on to the scene. The first large language model they trained was Gopher on this massive text data set which was-- the Gopher model actually was not very, very good. But the data set-- the paper does a great job of describing the data set. So MassiveText contains MassiveWeb, which I'll talk about a little bit later. There's C4 and then books, news, GitHub, Wikipedia. There's no details about how those were processed. And as I mentioned before, this is obviously-- is not reproducible. It's fine. You train on GitHub, but how exactly was the data processed? So MassiveWeb, you kept English. And here again, like CCNet, they use quality filters that were based on manual rules. And they had rules which you'll implement in your assignment that look at things like 80% of the words has to contain at least one alphabetic character. They also used Google SafeSearch for toxicity filtering. And I think in those days, one of the main arguments for using manual rules is that you didn't want it to be biased against the model, because models-- the only models that they can run is very weak models.

43:35 And weak models don't really understand the page. And they're just going to have probably pretty awful bias. And also, there was a consideration that this type of filtering can filter out kind of marginalized data from marginalized groups that didn't look exactly like Wikipedia. But as you see later, this has flipped. And now, everyone's doing model-based filtering. So their data set was 10 terabytes of text, which is, I guess, maybe let's say four or five trillion tokens, just an estimate. Although Gopher was only trained on 300 billion, which is not very many tokens. I think that's around the same number of GPT-3. So in 2022, we have Llama. So the data set for Llama was Common Crawl process with CCNet. And here are the classifier here. There's a subtlety here. Remember GPT-3 was classified whether it looked like a Wikipedia page or not. Llama was trained on classifier which predicted do you look like a page that was referenced out of Wikipedia or not. And I guess the idea is that Wikipedia will cite high-quality pages. And most of those pages might not look like a Wikipedia article, but nonetheless are high quality. So again, this link structure which we saw in the GPT-2 WebText is showing up here. They also included C4.

45:17 Why not? They use GitHub. They kept permissive licenses filtered based on some manual rules, Wikipedia, Project Gutenberg, and Books3, which got them in a lot of trouble, and arXiv, Stack Exchange. So they got 1.2 trillion tokens. So they didn't release the data set, but together reproduced this data set and something called RedPajama, which now you can go and-- you have the data processing code and the data, which you can take out. So this was a reproduction. So this was clearly not optimal. And Cerebras did further deduplication and ended up with a 627-billion token subset. There's also RedPajama-V2, which is a little bit confusing because this is something else. This is essentially taking Common Crawl snapshots and producing 30 trillion tokens with all sorts of different quality signals. So this is a resource for doing research on how to filter based on the quality signals that are computed. OK, so that was Llama. And then the RefinedWeb was another paper. And here, the thesis is-- well, remember how we saw the Pile. There was web data, and there was all this other stuff. And their point was like, well, maybe if we do a good enough job filtering the web data, that's all you need, because technically, the internet has everything in some sense. If you think about it, if you can access it via computer

46:58 and it's connected on the internet, then maybe that's good enough. So the RefinedWeb-- let's see if we can look at some examples here. The data is on Hugging Face, and it kind of looks like this. The resolution is probably not large enough to-- OK, anyway, scrap that. They used Trafilatura for extracting content because as we noted, Trafilatura is better than just using the WET files that Common Crawl provides. They used Gopher rules. And they made a point of we're going to avoid ML-based filtering to avoid biases. And then they did some fuzzy duplication. So they had a 5 trillion token data set, which is quite large, but they only released 600 billion of it. FineWeb from Hugging Face was started as a replication of RefinedWeb, but they tried to improve it. So they used all the Common Crawl dumps I think at the time, did some filtering. Again, this is still using manual rules, no model-based filtering. And they did some deduplication and did some basic anonymization. So they got 15 trillion tokens out of this. So this is still, I think, a really nice data set because it-- dealing with Common Crawl is going to be a pain, but this is a-- I would consider FineWeb as a lightly filtered data

Dolma, DCLM, Nemotron-CC: classifiers win

48:45 set that you can further do model-based filtering on. So jumping ahead, so AI2 has a series of models called OLMo. Their initial model was trained on the Dolma data set. And this is a composition. So we have Common Crawl. We have The Stack, which we talked about, which has code, C4, which you know, Reddit. AI2 has Semantic Scholar. So I think this is derived from that, Project Gutenberg and Wikipedia and you know about. So the Reddit comes from this project, but they include the submissions and the comments separately. So you don't have this thread structure. I think this project-- I don't know if it's still-- I guess it's no longer exists anymore. Around 2023, all these sites like Stack Exchange and Reddit realized, wait, people are just taking our data and training models and making money off of it. So I think that came to a stop. So we have 40 million academic papers from Semantic Scholar, which crawls a bunch of different sites, and then we have our usual suspects. So the Common Crawl processing, which is fairly, I think I would say standard. So they use language identification to keep only the English part quality filtering. Again, in Dolma, for training an initial model,

50:35 they avoided model-based filtering. And then toxicity filtering, they hear you, they use a classifier. And then they do deduplication. So 3 trillion tokens came out of that. And then in the same year-- so this was last year. So there was a paper DataComp which was a collaboration from multiple different organizations. Here, what they wanted to do is foremost define essentially a competition for creating data sets. And so they wanted to set up basic infrastructure. So they define a standard data set, which you can essentially try out different data processing algorithms. So they processed Common Crawl, all the dumps to produce DCLM-pool, which has 240 trillion tokens. So that's a lot of tokens. But as you know, Common Crawl is not the highest quality on average. So that's going to get filtered down quite a bit. They had a particular recipe for filtering down that data set, which is called DCLM-pool into DCLM-baseline. And here, they were very aggressive in using a quality filter. So this is what it looks like. They do some rule-based filtering, basic stuff. The main thing that's interesting is that they took this fastText filter that filtered DCLM-pool into DCLM-baseline. So they only kept, I guess, 1.4% of the total data set. So what do they do for this model based filtering? So again, in quality filtering, you

52:24 define your positive examples, negative examples, and you train a classifier. So the positive examples come from two sources. There's this OpenHermes data set. So this is mostly GPT-4-generated instruction data. So this is interesting. They're using actually instruction data to curate pre-training data. So now, they're not explicitly in training on instruction data, but they're looking for data that looks like instruction data. And then ELI5, which is basically the subreddit called ELI5, ask me like I'm 5. And this is what I guess the data set looked like. What's the point of wasting the first two plays with a rush? So these are questions you might ask to a chatbot, for example. Negative examples are just sampled from RefinedWeb. It's not low-quality data, but it's not as curated as these other two sources. So the result is that they took DCLM-pool, which is 240 trillion tokens, and reduced it to 3.8 trillion tokens, which is still a good sizable chunk. So then they trained a fastText classifier on these and run it on all of DCLM-pool. And here's one of the results tables. So the benchmarks here are core includes a bunch standard language modeling benchmarks like HellaSwag and so on. And they show that using this classifier,

54:16 they actually outperform the RefinedWeb by 3% and a bunch of other things by 1% or 2%. So this is the procedure they used to create this DCLM-baseline, which then you train a pretty reasonable model from there. It is worth noting that after this happened, the second OLMo model, if you remember, they started training on DCLM-baseline as well. So I think this era of we're going to be unbiased and try to not use models to bias, I think has kind of largely gone away because I think people realize that, well, if you use models in the loop, you can just do a much better job of getting high-quality data at least for increasing benchmark scores. So the final pre-training data set I'll talk about is Nemotron-CC. So this came out of NVIDIA, which has been doing some work on training the Nemotron models, although more recently, they've been doing post-training and data curation. So their main thesis is that DCLM, the baseline is great data set, but it filters very aggressively. Remember, it filtered out all the way from 240 trillion tokens to 3.8 trillion tokens. And if you want to train larger models for longer, you need more tokens because this aggressive filtering 3.8

56:01 trillion isn't really enough to sustain, let's say, a 400-billion model training run. So the first thing interesting they realized is that they did ablations for HTML to text, but not just based on the quality but on how many tokens were left. So they're really trying to get not throw away tokens. And it turned out that jusText rather than Trafilatura would actually keep more tokens. So they went with jusText. And then they used a bunch of different techniques to do quality filtering. They prompted the gigantic Nemotron model to essentially score documents based on educational value, distilled it into a faster model, and use that to train. So this is a filter based on what a language model thinks is educational value. And they also use the DCLM classifier. There's an interesting way that they ensembled. They basically ran all these classifiers and bucketed their scores. And then from each bucket, they sampled a bunch of data set. So it's not just taking the top, because I think they're trying to make sure they have good coverage over different experts kind of opinion on the notion of quality. They also did this thing, which is interesting. They used a language model to not just filter but also rephrase data sets. So for high-quality data sets-- sorry, this is backwards. For low-quality data sets, they basically use a language model to rewrite it into something that looks higher quality. So obviously, there could be mistakes there.

57:43 But in the grand scheme of things, maybe it's no worse than training on low-quality internet data. And for high-quality data sets, they use the language model to generate things that look like tasks. So you take a Wikipedia article. You ask a language model to create-- essentially input output pairs, where the input might be a question and outputs an answer, or the input might be summarize this document and outputs a summary, or the input is extract the key information and the output is key information. So this is, again, trying to get at the idea of, well, eventually an instruction to any time we want to be able follow instructions. So we might as well get a head start on this. So they got 6.3 trillion tokens out of that, which is more than 3 trillion tokens, almost double, which is I guess, pretty good because I mean, all of this is coming from Common Crawl, but they were able to essentially double the-- maybe not quite double, but almost double the size. For reference, Llama 3 is trained on 15 trillion. Qwen3 is trained on 36 trillion, which includes, I think, multimodal data. So 6.3 trillion at this point is not enormous. But for open-source models with data sets, it's pretty decent. Most of us, 6.3 trillion is more than enough for training, even to do one epoch. This table shows that basically, on average,

59:22 they show that their Nemotron-CC data is better than the DCLM data, which has been shown to be better than FineWeb, at least on benchmarks. And then they actually have this 1 trillion high-quality subset that's even better. So any questions before I move on? I know this was a lot of random specific details about different models or data sets, but hopefully, this gives you a sense of the type of things. And hopefully, you can see different patterns,

Interlude: what about non-English data?

59:56 whether to use models, to filter or not, whether you're using links out of high-quality pages or using the pages themselves, and so on. So yeah. Let's talk about English data sets. I was wondering if you come across things like multilingual data sets, that kind of thing. Yeah, good point. So the question is, what about multilingual data sets? I've focused on English because that's where primarily a lot of the research is done. But obviously, Common Crawl does have multilingual data.

Copyright: licences, fair use, terms of service

60:31 And there's multilingual data sets that are produced as well. OK, maybe interest of time, I'll move on to copyright. So this was an early question that was asked, how much of the web is copyrighted? So let's just understand what copyright is really about. So nowadays, there's a lot of lawsuits around generative AI mostly around copyright. So in general, the copyright law falls under kind of intellectual property law. And the goal here is to incentivize the creation of intellectual goods. That's why copyright law even exists. And there's many types of copyrights, patents, trade marks, trade secrets. So copyright law is the one that has been-- is most relevant for training data. And this goes back to the 1700s in England. But in the US, since 1976, there's been the Copyright Act, which has-- essentially, it's established what copyright means. Formally, it's applied to original works of authorship fixed in a tangible medium of expression. And so it's original work. So if you are just a collection, it's not copyrightable. Like telephone directories are not copyrightable unless there's some creativity in their selection or arrangement. So copyright also applies to just expression, not idea. So you can't copyright an algorithm. You can copyright the code. And one thing that has been--

62:14 what this did is that-- before, copyright only applied to things that had been published. And now, it's just this looser notion of fixed. In general, copyright has increased in scope. And registration is not required for copyright. So this is different from patents. If you invent something and you don't register, then you have no claim to it. Whereas copyright is you put something and you throw it up on your website, it's copyrighted even if you don't explicitly write copyrighted. But there is a thing, that registration is required before a creator can sue someone for copyright infringement. But the bar for registration is also only $65, as opposed to patents, which can be thousands. It now lasts 75 years, and then the copyright expires and becomes part of the public domain. So all the classics and most of Project Gutenberg has been gone out of copyright. So the thing that maybe people might not realize is that most things on the internet are actually copyrighted. So whether something copyrighted isn't really the issue. So the issue is whether you can use it. And there's two ways you can use it. You can either get a license for it or you appeal to the fair use clause. So if you're going the license route, you can either go sign a contract with the creator to get a license to use the data in some terms.

63:56 And this is essentially what does with-- for example, Google and Reddit have this relationship. And effectively, license is don't sue me. There's a special type of license called the Creative Commons license, which allows the free distribution of copyrighted work. So Creative Commons, all that stuff is still copyrighted. It's just that you have a license that enables it to act like if it was in the public domain. So a lot of things are-- like Wikipedia, for example, is all Creative Commons license. And a lot of YouTube videos are also Creative Commons. And this was created almost, I guess, 20 years ago, to essentially bridge the gap between public domain and copyright. And the idea is that you want people to be able to use this stuff without waiting 75 years. And there are cases where the creator is actually happy for people to use their content, but most of the time, it's just not clear because they haven't said yes or no. So now, many model developers license data for training foundation models, for example, Google and Reddit, OpenAI and Shutterstock, OpenAI and Stack Exchange, and so on. So if you have money, you go get a license. If you don't, then I guess you have to say you're a poor academic, and then maybe they'll let you use it. But the problem is that you can't get a license to the internet. You go to a random website. Who do you even go talk to? So the only way you can use it legally

65:41 is to appeal to fair use. So fair use says basically, even if something is copyright and you don't have a license, you can still use it under some conditions. And the conditions are determined by the purpose and character of the use. So for example, if you're using it for educational rather than commercial, or you're transforming the work in some way rather than just like copying it and hosting it on your website and pretending it's your own, that's going to help you. What the work is, if it's fictional, it's more likely to be-- actually, if it's, sorry, factual, it's more likely to be fair use. For example, the telephone book. You can't really copyright things that are closer to facts. If you use this just a snippet, then it's more likely to be copyright fair use. Although for language models, that doesn't really apply because you probably want to train on all of it, not just a snippet. And then effect on the market. So for example, if you're using the work to essentially displace the creator, then that's going to be seen less favorably than if you're using the work to do something completely different. So if you obviously watch a movie and write a summary of it, it's fair use. If you reimplement the idea, that's fine. There's a big, long-decade fight over whether Google Books when they show snippets, whether that's fair use or not. And eventually, it ruled in favor of Google. It's also worth noting that copyright isn't just about verbatim memorization. So it turns out that plots and characters can be copyrightable. So even if you have essentially very little n-gram overlap,

67:34 but you take Harry Potter, the character, and you essentially develop it, then that could be a violation of copyright. But on the other hand, if you parity, it might be fair use. So these things are quite subtle. Copyright is all about the semantics and the economics and the kind of content. So it's a very complicated topic. So what about for training? So one thing is that copyright is-- copy is in the name copyright. The first step of training, you're copying the data is technically already a violation, even if you don't do anything with it. You could argue, as many have, that training ML model is transformative because it's definitely far from just copy and pasting. This does make, I think, open-source models with open data bit challenging, because if you want to train a model and you also want to show people your data and you host that data, that could be in violation of copyright. People have also made arguments that the machine learning system is interested in the idea, not the expression. You're training all this data because you're trying to extract how language works and general knowledge, rather than interest in a particular work. But of course, the learning algorithm has been-- sorry. The models can often memorize, and you can extract training data from the models quite easily.

69:17 And there's also this problem that language models can definitely affect the market regardless of copyright. One, I guess, other thing to note is that even if you have a license and if you can appeal to fair use for a particular work, you still might not be able to legally get the data because of terms of use. For example, YouTube has a lot of Creative Commons videos, but if you write a script that goes and downloads your videos from YouTube, that is against the YouTube's term

Mid- and post-training: long context, tasks, instructions

69:51 of service. So there's another gating for these platforms. There's a bunch of works that you can read about later. OK, so let me quickly go on in the interest of time. So this section is going to be a bit shorter. And I've collapsed mid-training and post-training together because the boundary is not often quite clear. Often now, we're thinking about less high quality in general, but focused on how do you instill particular capabilities, although even in pre-training, we were already thinking about quality classifiers and high quality. So again, the line is not quite clear. So one thing that I think is-- which we haven't really talked about in this class, is long context. So models, if you look at the top models, have quite a bit of context. Gemini I think still has-- I think Llama 4 might advertise a 10 million contexts, but the context lengths are quite large. And transformers scale quadratically with the sequence length. I mean, we saw the inference lecture, you can get around that, but still you need, I think, full attention to get the best results. And clearly, you don't want to start at the beginning training on long context. So what people do is they add it later. So that's why long context extension often shows up at mid-training because you don't want to waste cycles training on long contexts if your model is not very good. So there's multiple ways of doing that. But since this is a data lecture,

71:34 I'll talk about books and math as our two sources that have been used to do context extension. Basically for context attention, you just need to create data that has long range dependencies. And some of this data can also be synthesized. So people also look at tasks. So there's a bunch of works that essentially convert traditional NLP benchmarks into a standard format that they can be fine tuned on. So Super-Natural Instructions is one such data set that leveraged the community to come together and create 1,600 tasks, and they standardize into a prompt. Flan was around the same year. It came out in 2022, but the paper was in 2023. So 2022 was the year of let's take all the NLP tasks and shove them into instruction-following format. One, I think, advantage of this is now you have a language model that can solve all your favorite NLP tasks, and you benefit from transfer learning. This is a lot of thinking about-- going back to T5. But one problem with this is that often, the prompts that you have are very templatized. If you look at Super-Natural Instructions, some of it is not supernatural because they're all look kind of the same.

73:24 So that motivates these instruction-following data sets. And since 2022, there's been this expectation that language models should just be able to answer any one-off task that you give it. So the notion of even tasks disappears. So a lot of the work in the open community has been based on synthetic data, starting with Alpaca, which used this idea of self-instruct to prompt a language model to generate examples, which you can then use for fine tuning. There's Vicuna, which used these conversations that had been shared by users on ShareGPT, which is deprecated now. You can get language models to chat with themselves, seeded with some questions, and that creates some synthetic data. And you can also have these Evol-Instruct methods that essentially take questions and make them more complicated. There's other ways of taking-- this one takes Common Crawl and then uses it to essentially look at-- identify quiz sites and then extract QA pairs using a language model. And then this is an OpenHermes, which we saw earlier from the DCLM work. It's just an agglomeration of different data sets. Llama 2 chat, we don't know what the exact data set is, but they used annotators to essentially write high-quality instruction data. And they claim in this paper that this

75:15 was better than using the millions of examples from open data sets. But they could have even saved more money by less annotation and more just RLHF, which we'll talk about later. And finally, the last stage that I'll mention just came out pretty recently, so the Llama-Nemotron post-training data. This consists of a bunch of-- there aren't that many details about this data set, but the data set released, so you can go and look at it. They have public data sets like WildChat. And then they synthetically generated some data from all the models that you're able to generate data from. They also include reasoning traces, thanks to R1. So if you look at this data, you can divide it into a few buckets. One is that a lot of the early work was just, OK, there's GPT-4. This is the easiest way to generate synthetic data. The problem with that is that-- for academic research, it's fine, but it is against the terms of OpenAI to use GPT-4 to create a data set that you train a competing model. Whereas these open-weight models have more permissive licenses, which means that you can essentially distill from them and do whatever you want.

Summary: data does not fall from the sky

76:57 There might be some restriction on Llama, but I think broadly speaking, I think they're more permissive than OpenAI. And then finally, if you are really paranoid, then you can actually just hire annotators to create high-quality instruction, which is obviously more expensive and slower. And there's also this worry that annotators might actually use GPT-4 to create your data. So you have to be careful there. To summarize, so the key lesson is data just doesn't fall from the sky. You have to really work hard to get it. And it's important to think that out there in the world, there's these live services like GitHub. And first, you have to use it. You have to first get a dump of the raw data. But you can't train on the raw data. It's too big or it's too noisy or it has-- it's not even tokens. And you often have to process it. And that's where a lot of heuristics that we saw for quality filtering and deduplication kind of fit in. This was touched on earlier. Data is really the key ingredient that essentially differentiates language models. I think all of the language model architectures, there's some transformer MOE style. These architectures is so of general purpose that the behaviors aren't really going to be that different. It's really the data that drives the quality there, assuming you can train and fit the data. There are some legal and ethical issues here.

78:41 We talk about copyright, but there's much more to be said here. And then finally, if you think that this whole field is a mess, you're right. It's very heuristic, which means that there's many opportunities to hopefully improve that. OK, that is it. And I'll see you on Thursday.