Data 2

Where lecture 13 left off, and the plan for this one

00:04So this is the second lecture on data. In the last lecture, we talked about different data sets that were used to train various language models. We did a historical overview from the data sets that were used to train BERT to all the way up to Dolma and everything in between. And one of the things I wanted to emphasize is that data doesn't just fall from the sky. It exists often in live services. They have to be explicitly crawled or dumped. And then there's a bunch of processing that has to happen. And the processing involves converting the raw HTML into text, doing all sorts of quality and language and toxicity, filtering, deduplication, and so on, and so forth. So in this lecture, I would like to take a deep dive into some of the mechanics on how quality filtering and deduplication work in particular. So first, we're going to look at different algorithms for filtering, which mostly are model based. So you train a classifier or train some sort of model to filter. Then we're going to show that this primitive can be used for all sorts of different types of filtering. And then finally, we're going to look at some algorithms for deduplication.

The filtering primitive: given a little of T, find more of it in R

01:14So this lecture will be a little bit different in that it's going to focus a lot on classical big data processing algorithms. And there'll be some fun math in there for some of you. OK. So let's get started with filtering algorithms. So the basic, high level picture here is that you're given some target data, which you don't have very much of it. Suppose it's data that's high quality, and you have a lot of raw data. So imagine, this is Common Crawl. And your goal is you want to find a subset of that raw data to call it T prime, which is similar to T So this is the pattern that you should recognize in basically any filtering pipeline. And what do you want for this filtering algorithm, you want it obviously to generalize from T. You don't want— you already have T. So there's no point in getting exactly the same data T. So there has to be some generalization. And also has to be extremely fast.

KenLM: Kneser-Ney n-grams as a quality scorer

02:20So you have to run it on all of the web, for example. So if this is running a huge model, that's going to be— might be as expensive as training. So you definitely don't want that. Otherwise, you might as well train. So we're going to look at three different ways that you can implement this high level abstract primitive. So we'll start with training an n-gram model. So this is something that came up in the last lecture. So you can train an n-gram model with Kneser-Ney smoothing. You can use other forms of smoothing, but Kneser-Ney has been inherited from the. N-gram kind of modeling statistical language processing era. And there's a particular implementation that people tend to use in this space called KenLM, which is open source, originally developed for machine translation, which implements Kneser-Ney smoothing. So you can go and read about Kneser-Ney. It's pretty interesting, but we're not going to talk too much about the details here. So this happens to be very popular in data filtering. There's no particular reason it has to be this one, but this is just what people use. And the nice thing about n-gram modeling is it's very simple. When I say fitting a model, you should think of this as just counting the number of n-grams, and then normalizing. So just to go into a little bit of depth here. So your starting point is maximum likelihood estimation of language models. So you have a corpus of text, and you

04:00want to estimate conditional probabilities, like p of n given the last n minus 1 words, the cat. And what you do is you count the number of n-grams, and then divide it by the number of the n minus 1 grams of the conditioning context. So this is very, very easy in principle. Now, implementing this efficiently takes a bit of work. But the main problem is that there's sparse counts. So many of the n-grams are exactly show up zero times even though they're very reasonable, especially when n grows. And this was the whole point why n-gram models couldn't really be scaled appropriately because you hit the curse of dimensionality. So to mitigate this, some people devised Kneser-Ney smoothing to handle unseen n-grams. And there's more details on this, but roughly, this says, well, maybe if I don't have enough data to support this count, I can use a lower order n-gram. So remove one word from the context and count the number of times in, appears given cat, and then try to either interpolate or back off to that. OK. So that's all I'll say about how n-gram modeling works. I think even in this era of neural language model, it's good to know how n-gram models work. So let's download the n-gram. So this is a model that's been trained on Wikipedia. I'm just going to download it.

05:40And then let's punch some sentences through. So this sentence is Stanford University was founded in 1885. This is actually an excerpt taken from Wikipedia. So it should be reasonable. And if you compute the probability, so you first compute the log prob, and then you compute the perplexity by this normalization, then you get 187. OK. So it's some number. And then here's another example. This is taken from the CS 336 website. So let's see how well it does. So this is a higher perplexity, means that it's less likely. Kind of makes sense because this stuff doesn't really probably show up in Wikipedia. It's not in— it's not abnormally high because it's still fairly good well-formed English. And many n-grams there, I think, show up in Wikipedia as well. What about ASDF? This is just gibberish that gets assigned higher perplexity. And then there's da, da, da, da. So this should get assigned pretty high perplexity. But for some reason, this particular n-gram assigns a pretty low perplexity. So it's an n-gram model, so it's not particularly clever or good. But you don't need a P the best model. You're just doing data filtering to create a model. So this can be very crude and fast. So CCNet, which is a paper out of—

07:21I guess it was a Facebook back then. They— which we talked about last time. They basically did this and actually— OK, that's fine. So they basically looked at paragraphs of text. And those were the items, they sorted the paragraphs by increasing perplexity and just kept the top 1/3. And this is what they used to create the first LLaMA dataset. OK, so this is heuristic, but if you had to write down something really quickly, this kind of makes sense. So Kneser-Ney language modeling is fast, but it's crude. All right. I feel like there is some content that was meant to go through that's missing. Oh, well, maybe it got deleted. All right. So you can fit an n-gram model to your target data, and then you can use that to score the raw data

fastText: a linear classifier over hashed n-grams

08:34and choose the top scoring documents. Another thing you can do is fastText. And this is, in some ways, more popular. This is also from Facebook. They released a lot of open source tools, although KenLM, I think, was created before that. So here, they develop it for text classification. And this paper, I mean, this is 2016. So a lot of people were developing all sorts of fancy neural models. And they said, show that well. This actually— almost linear classifier worked as well. It was much faster. So that's why the software package actually became very popular. So the motivation here was that if you're doing, let's say, a bag of words classification. So you have a sentence of length L, your vocab size is V, and you want to classify into 64 classes. Then immediately, you have to define this V by K matrix, which if V and K are both large, then this gets pretty big, and you get into sparsity issues, and the way it would work is that you take your documents, and then you just do linear classification. So that's fairly straightforward. So the problem is that the number of parameters is huge. So fastText says, well, let me just do a linear— basically dimensionality reduction. So instead, I'm going to map my vocab space into a hidden dimension, which is smaller than K, possibly smaller than K, 16, and then I'm

10:20going to classify from H. So notice that when you look at the prediction, the forward pass, there's no non-linearity. It's just a linear classifier. You can also think about it as a matrix factorization if you want. And the number of parameters is greatly reduced. So the implementation, I think, is reasonably optimized. It's parallelized. And they use asynchronous SGD. So here's— OK. So the question is, this is bag of words. And we want to not just look at words, but we want to look at n-grams. So this paper also extends to n-grams in a fairly simple way. So you take a sentence, and you get some— chop it up into n-grams. And here, the problem is that the number of bigrams can get quite large. And it actually can be unbounded. So remember, when we talked about tokenizers. If you just chop up into words, you're going to get— You know how many words there are. So the simple thing is that you just do hashing. So you define some number of bins, maybe 10 million. But in this example, eight. And then you just hash every bigram into that. So the cat maps to two and cat in maps to one, and so on. So of course, there might be some collisions, but just live with that. I mean, in some sense, when you minimize the loss, collisions are accounted for.

12:05If there's two words that have nothing to do with each other, then the weight is going to somehow represent maybe the average of the two. And also note that often, we use fastText classification with K equals 2 classes, is this document good or is it bad? And in that case, this is just normal. Basically binary classification, linear classification. So of course, it doesn't take too much imagination to imagine. Well, if you can do linear classification, you can use fancier models, such as BERT or even Llama. The only thing is that it's going to be slower. And therefore, you run into this trade off, where if you use a giant model to do classification, maybe you should use those flops to actually just train the model. So that's the only concern. Because the web is very large. And remember, you were filtering the web down quite a bit. So you're going to spend a lot of compute on, let's say, you're filtering down to 1% of the web. So that means you have your classifier.

DSIR: importance resampling instead of classification

13:14That amount of compute you spend has to be 100th of taking a forward pass through the data set. So the final method I'm going to talk about is this thing called data selection for language models using importance sampling. And here, the basic idea is that you have your R. It's a raw data set. You have your target data. And then I'm going to build an importance weight estimator, which is the analog of the language n-gram model or the fastText classifier. And then I'm going to use that to get a bunch of documents. So first, just a kind of a quick refresher of importance re-sampling, I guess. So you have— this shows up in many contexts, like particle filtering and Monte Carlo methods. So you have a target distribution. So it's a distribution, not a data set, where you want to ideally draw samples from this. But you only have a proposal distribution Q, where you can draw samples. So for example, if you have a vocab of 0, 1, 2, 3, let's say this is your target distribution. And your proposal distribution is this. Now, what do you do is you sample from Q because that's the only thing you can sample from. So let's say you get a bunch of samples. So notice that Q has a lot of more mass on zero. So you get a lot of more zeros.

14:58And then what you're going to do is you compute weights over your samples by looking at this importance ratio, p over q. So you're trying to make up for the fact that I sampled from q. So I want to divide that out or I really want p. And you normalize and then get— you get a distribution over your samples. And now, you resample, and you— that balances out the distribution. So now, you see that there's more 3s because p had more probability distribution on mass on 3. So this is a fairly elementary, but it's a building block. Now, we go to how we do data selection. So we have a target data set, not a distribution. We have a data set, which is small. And we also have this raw data set, which we're going to call the proposal data set, Dq, which is large. So you could— let's just fit a distribution p to Dp, theta distribution q to Dq, and do the importance sampling that I we talked about just two minutes ago. Now, the main problem is that Dp is really small. Remember, target distribution is high quality data. You don't have that much of it, which is the whole point. You're trying to get more of it. So you don't have that much of it. And it's too small to estimate a good model. So again, the idea is you just use hash n-grams. So this is something that we already saw with fastText as well.

16:41You take whatever text you have, and then you essentially— so you basically hash all the— we're doing unigrams for now. So you hash each unigram. And you basically count the number— estimate the probability of every hashed n-gram. So in this case, the cat in the hat hashes to this. Notice that there's some hash collisions though, and the hat hatch into the same value. But whatever. This is all crude. And then you just estimate the probability of each of these— the probabilities. OK. And then to evaluate the probability of a new text, you hash the new text, and you multiply the probabilities together. OK, so I got a little bit unlucky, and the probability gets zeroed out. You can do some smoothing if you want. So it turns out in Python, hash is non-deterministic. So every time I run this, I'm going to get something else. And the paper shows that this indeed helps a bit. The games are not super massive. But compared to doing fastText, which is the heuristic classification, the gains on the GLUE benchmark using BERT style models. There is some lift. And the comparison with fastText. So the main idea here is that modeling a whole distribution is, in some sense,

18:24a more principled approach. Because you want to sample data from your distribution, and you have a different distribution q. And now, you're trying to essentially compensate. And so this could be better for diversity because you're trying to match the distribution. Whereas fastText, you're essentially trying to classify whether something is in the distribution or not. And it doesn't have any guarantees about matching the distribution. But it's also fast like fastText.

One framework, three scoring functions

18:59And both can be improved by having better models. So instead of using linear models based on n-grams, you can increase n. You can use neural approaches as well. OK, so just to wrap up this part. So we looked at n-gram models, linear classifiers, and importance sampling. But the general framework, I think, is something that maybe worth taking away, which is that the setup given a target data set and a raw data set, find the subset that's similar to the target. And all these methods have the basic recipe, which is estimate some of model based on the raw data and the target data set, which gives you a scoring function, and then keep examples in the raw data set based on that scoring function. Usually high scores are kept. So just to walk through this in the KenLM case, the score is simply the probability of— or I guess maybe the perplexity, if you want to normalize by the number of tokens, under the target. So r is not used here. And then you keep examples with score greater than threshold. And you can randomize a little bit around the threshold if you want. So you can also use a discriminative classifier, where the score is what's the probability that this example came from the target as opposed to the raw data set. And then you keep examples, where the score is

20:43of greater than some threshold. And then importance sampling says the score is the ratio of two generative models, which could be n-gram models in this case. p of— the target distribution over the raw distribution. And to use this score, this is really the importance weights. You resample with probability proportional to that. So at a high level, all of them are doing essentially the same thing. You have— you fit some of distribution that tells you what looks like target as opposed to raw, and then you apply that to the raw data set. OK. Maybe I'll pause here in case there's any questions. Yeah. It's sort of a high level philosophical thing. We have an idea of what good data is, but we've kind of not talked about what that's actually like. Like a good document looks like. So what this is doing is like making sure that the words fit next to each other when you have an environment like that, one comes after the other. But it doesn't ensure that it makes any sense on like a larger level or if you want that just increase the n and the n-gram? Yeah. So the question seems, I think, is that if you're using an n-gram model, it's only looking at local contexts. And maybe that's not great for assessing quality.

22:26Because if you shuffle everything around, it still looks good to the n-gram model. So there's definitely that danger that it's very easy to adversarially gain. If you can construct, you can definitely construct examples that the n-gram model thinks is high. I mean, I showed you one that does happens to be pretty high. But somehow, on average, you're doing OK. And in some sense, maybe if a document looks like grammatical English, and it looks like news or some scientific paper, all you want to make sure is that that model assigns high probability to that. If you get some bad examples, it's not the end of the world. Yeah. So we should think about these as just filtering out true nonsense web pages that you— Yeah. I would say this is filtering out true nonsense is I think a good way to look at it. As I go through some examples, I think

Language identification, and OpenWebMath as a case study

23:31maybe it will become a bit clearer that this is— these classifiers are not just used for quality. And for some of the other use cases, I think it's much more maybe compelling why it should work. OK. So the same data filtering machinery can be used on different tasks. So I'm going to talk about language identification, quality filtering, and toxicity filtering. So language identification, the idea is that you want to find text of a specific language. Maybe it's English. Maybe it's Japanese. And you could ask, well— you could just train on all the languages. Why don't you do that? The problem is that it can often be difficult to do the curation and processing of high quality data in a given language. And there's also the problem that if you train— let's say you only care about one language. Now, if you have data from all the other languages, you're going to be spending less compute and tokens on any given languages on the given language. So this was— and that could hurt your performance in that language. For example, BLOOM, which was trained in about 2022, I believe, was only trained on 30% English. And as a result, the English performance maybe was not as good as it could have been. It was strong on other languages. So there, in a compute limited regime, you're definitely at risk of compromising the quality

25:14of the language you care about. Now, that said, if you have enough capacity and you're training huge models, then you can definitely train multilingual models, and there could be even positive transfer between languages. So all the kind of frontier models are trained on heavily, multilingual, sometimes even hundreds of languages, like 100 plus languages. So one simple way to do this is fastText is a toolkit for training simple classifiers. It also comes with some off the shelf models. And one of the ones that typically gets used is they have a language identification model. And this is just off the shelf classifier. It supports a lot of different languages. And it was trained on basically multilingual sites, including Wikipedia, which has a lot of different languages, as translation sites, and for some reason, Southeast European news. So Dolma uses this. They basically run the fastText classifier and keep pages with English probability greater than 0.5. So let's try it out. So there's this model language ID that you can just download. And then let's see how well it does. So the quick brown fox jumps over the lazy dog. Let's see where are the predictions. OK, I don't know why—

26:57OK. There we go. So— OK. So this says English. This is the label, English. And the probability of English is 0.71. So I mean, it would have guessed it would be higher. It looks pretty English to me. If you duplicate the sentence, the probability isn't changing, which is good. Saying the same thing doesn't make it more English. There's some informal English, which this is kind of weird. This is, I guess, more English than the quick brown fox. German. Actually gets classified fairly. Oh, sorry, this is German. Now, the label is German. OK, here. That is German. Math gets classified pretty weakly as English. Code is highest is, like Russian. This is— hello is apparently Italian. This is French. Bonjour. That's good. And this is— I guess, a tricky one, where I put in some Spanish, and also some English. And I guess it favors the Spanish in this case. OK, so this gives you a sense— I mean, it's always good to play with these classifiers. Just because it's a classifier, you can download, and everyone uses, it doesn't mean it works all the time. So I think obviously, for short sentences, it's less reliable because there's just less information about what's happening. Low resource languages is tough. There's a worry that for things that don't really— dialects of English, this could be viewed as not English. I think if you want to distinguish

28:42between similar languages, that also can be easily confused and code switching. I don't even know what ground truth is, but it'll do, hopefully, one of the languages. So just talking through one case study, OpenWebMath was this paper from two years ago. And the goal here is to curate a large corpus of mathematical text, where here, I'm saying that— let's assume that math is a language. So first, they use some rules to filter. Then they train KenLM on this large data set of proofs, called ProofPile, and kept it if the perplexity was below some threshold. And they trained this fastText classifier, which we have been— we've talked about to predict mathematical writing. This is a hacky, but they set the threshold of the classifier. If it's identified as math based on rules to be relatively low. And if it's not identified as math according to rules, it's needs to be higher. And as a result, they produced 15 billion tokens,

Quality filtering: GPT-3, LLaMA, phi-1

29:52and they trained some models that do better than models that were trained on 20 times more data that weren't really specified. So this is one of a nice examples of the utility of data filtering. You could train on all this data. But if you care about, let's say, a particular domain like math, you can just go out, and get more of it, and target that data set. And if you train on that, you can be much more data efficient. So let's talk about quality filtering. So quality filtering, obviously, is something that doesn't really— I mean, it's a catch all for a bunch of things and what is quality. So remember that some papers explicitly don't use model based quality filtering. But more recent papers have just kind of given in and say, well, I mean, it just works a lot better. So GPT-3, we talked a little bit about this last time, trained a quality classifier by taking positive samples from the high quality sources that they had, or rather the non-Common Crawl sources they had. And negatives are just Common Crawl. They train a linear classifier and keep documents stochastically based on this score, which is something that just turns numbers, basically sharpens the threshold. And the first LLaMA paper, they used pages referenced by Wikipedia. So not Wikipedia as positives.

31:29Negatives are just sampled from Common Crawl, and then they keep documents that are classified positive. I don't think there was actually sampling of that, which is interesting. phi-1 is another paper that— their philosophy was interesting. They really wanted high quality data that looked like textbooks, and they wanted to train a very small model. So they trained on synthetic data but also filtered data. So for the filtered data, what they did is they took the Python subset of The Stack. Remember, The Stack is this pre-processed data set coming from GitHub of code. And then they define the prompt, which effectively asked GPT-4 to determine the educational value for a student whose goal is to learn basic coding concepts. And then they prompted GPT-4 and classified 100K documents from this Python subset. And that became the positive examples. They train a random forest classifier on these examples, where the embeddings fed into a random classifier were from some pre-trained model. And then they selected data from R that is classified by the classifier. So they run the classifier, then over all of R, and then get that data. So this is interesting because there is a random forest classifier. And in this case, where does T come from? Well, T doesn't exist a priori. T comes from prompting GPT-4 with some prompt. So this is something that you'll see more increasingly as the models get better, is that instead of looking

33:14at sources like, well, maybe Books1 is great, or maybe Wikipedia is great, now, if you have a good model, you can just ask the model for the type of data you want. Maybe I want more chemistry data, or I want more mathematical data that does a lot of proofs or elementary math or something, and you just let the language model a fairly sophisticated language model give you T. And now once you have T, you turn the recipe that we've been talking about so far. So the phi-1 results, again, looks good. So they had a baseline, where they just took the Python subset of The Stack, which was this R. And the performance on HumanEval, which is a coding data set was only 12% after 96K steps of training. And then they trained the exact same model architecture on this new fancy data set. And it was already at 17 after 36. So they declared success. So these are some examples of quality filtering. Any questions about quality filtering? Yeah. We can get away with using the most expensive model in the target data set using the LLM and not using LLM like you— [INAUDIBLE] Yeah. So the point is that you can— why can you get away with using GPT-4 here? Because you're using it only on to create a 100K examples. And to create— and then using that to distill a classifier, which is much faster.

Toxicity filtering in Dolma

35:00And the 100K is much less than the size of R, which is hundreds of— tens or hundreds of millions, probably. So let's talk a bit about toxicity filtering. So I'm going to talk about how the Dolma paper does toxicity filtering. So there's this data set called Jigsaw Toxic Comments. And this came out of a project, where they were trying to help people have better discussions online. So they wanted to build a classifier that could identify when there was problematic content. So they took the data, comments on the Wikipedia talk pages and had annotators annotate them with toxic, severe toxic, obscene threat, insult, identity hate. And so the Dolma folks train two fastText classifiers, one which is basically corresponding to hate, and the other one is corresponding to NSFW. So here are some examples of the data set. So let's download the model and try it out. So the first example, are you threatening me for dispute neutrality? This gets classified as safe for work. Because I mean, it seems fine. And the second one is flagged as NSFW. And these sentences are both fine. So just give you a sense.

Deduplication: exact and near duplicates, and what they cost

36:47I don't want to show too many of these examples, but you get the idea. You can play it out with your own time. So that's all I'll say about filtering. So now, let's talk about deduplication. So there's two types of duplicates, exact duplicates. And the thing to realize is that the web actually just inherently has a lot of exact duplicates based on mirroring. So if you just look at Project Gutenberg, you'll see that the same exact site gets mirrored on a bunch of different URLs. So if you're just crawling, as Common Crawl does, it has no way knowing that these are mirrors, and it just get you the exact same content. And then there's near-duplicates, which is basically the same text that differ by a few tokens. And we're going to show you algorithms that deal with both. So near duplicates, when does this happen? Well, you have terms of service and licenses, like the MIT license. This thing shows up I don't know how many times on the web. It's just copy and pasted everywhere. Maybe there's even— I don't know. Maybe people take it and actually change some things, or they messed up, copy, and pasting, and remove a comma or something. That's all possible. And then there's cases— if you look at these data sets, near duplicates are— it just gives you an idea of what near duplicates might look like.

38:25Sometimes, you have this article, where someone just seemed to have paraphrased best actor in a negative role for most impactful character, but everything else seems the same. This is just like missing a comma here. Not really sure where that comma went off to, but there could be just annotator processing errors. And then this one's kind of interesting. There's clearly this very ad text, which they just replace Canada with USA and some other flights. So this probably is generated from some of template system, where they slot it in entities. So these near-duplicates are obviously different. But obviously, if you train on tons of— once you have this document, this one doesn't give you that much value. Let's just put it that way. So in extreme case, so if you look at C4, remember, C4 was filtered based on these set of rules that looked for sentences that looked like English. You'll see this sentence, which is indeed English, that shows up an astonishing 61,000 times. And for whatever reason, if you actually trace down where this comes from, not the MIT license. Oh, I have to— what? OK. Should I accept these cookies? OK, fine. This is random product from Amazon that has graffiti mask drawings, and there's this text. And apparently, this is— for whatever reason,

40:12I don't even know how there's 61,000 copies of this, but that's what it is. And so the idea is that this text isn't necessarily bad. It's perfectly good English. But you don't want to take 61,000 epochs over this. I mean, it just kind of pointless. So deduplication is complementary to quality filtering. Quality filtering says this piece of data, I don't want to ever train on it. Deduplication says, well, this might be fine, but I only want a small number of these rather than 61,000 copies. So there's been work showing that deduplication makes language models better. The first, most obvious thing is that you can train more efficiently. Deduplication reduces the number of tokens. And if you haven't thrown away information, then you can save a lot of effort. And then the second is that you can also avoid memorization. So we haven't really talked about this, but language models trained on data, can memorize that data, which is bad for copyright or privacy reasons because it can just regurgitate the training set.

Design space, hash functions, and exact deduplication

41:28And deduplication is one tool. It's not definitely not perfect, but it can help mitigate some of these risks. OK, so here's how to think about deduplication in terms of design space. So first, you have to think about what is the unit that you're deduplicating. What is an item? Is it a sentence? Is it a paragraph? Is it a document? The second question is, how do you match? What is considered a duplicate? Is it an exact match? Is it a fraction of common subitems? And then the final thing is, what action do you take? Do you remove all the instances, or do you remove all but one? OK. So the key challenge, and this is a algorithmic challenge is that fundamentally, deduplication is about comparing items to other items. Quality classification, you can take an item, you can classify it as high quality or not. Deduplication is fundamentally a pairwise thing. And remember, these data sets are large. So you can't— if you're trying to do something quadratic, that's kind of a no go. So you need some of linear time algorithm to scale, but still do something that's pairwise. So the building block of all of this is hash functions. So hash functions, just as a review, is a function that maps an item, which could be a sentence, it could be a document, to a hash value, which is an integer or string. And the value is much smaller than the item. And the thing that's relevant about hash functions

43:09is that you could potentially get hash collisions, which means that you have two distinct items that map to the same hash value. So in general, if you've— I'm sure you've all— I guess all of you have used dictionaries or hash tables. Hash collisions are generally bad. But later, we'll see that there actually can be good in some limited doses. And there's many types of hash functions, which trade off between efficiency and collision resistance, which means don't collide. There's cryptographic hash functions, which are used for security sensitive applications, like in Bitcoin. You really don't want collisions. Because otherwise, that means someone might steal your money, or it could compromise things. And then there's things where— but these are slow. But then there's hash functions, which are generally used for hash tables that are not collision resistant but fast. And so we're generally going to go with the latter category because we want things to be fast, and we're not doing cryptography here. So we're going to use this thing called MurmurHash, which takes strings and returns values. But you can use a bunch of other hashes as well. So let's start with exact deduplication. So here's a simple example. The item is just a string, and I want exact match, and I want to remove all but one. So here's a bunch of items. And what I'm going to first do is compute a mapping from hash value to the list of items with that hash. And then once I have that, I keep one item from each group.

45:00So each group corresponds to a hash value, and in the case of exact match, assuming there's no hash collisions, then you basically do an exact dedupe. So hello shows up twice here, and now it shows up once. And of course, hello, uppercase exclamation point is just a completely distinct item because we're doing exact. So this is— the nice thing is that this is very simple. It's clear what you're doing. It's high precision. So you're never going to throw away anything that you didn't— that you didn't need or you needed. But the con is that you— it doesn't work for near-duplicates. And the other thing is that's nice is that this is very simple to implement. We kind of wrote this in a MapReduce way, and you can scale it and parallelize it fairly easily. And this is something that C4 uses, actually, to prepare their data set. So their unit is of deduplication is three sentence spans. They use exact match, and they remove all but one. So one thing that is always kind of bothered me, but I know no one else seems to care, is that when you're looking at three sentence spans. So you have this document, and then you have this three-sentence span. And if it's marked as a duplicate, you're just going to do surgery and take those sentences out. And so the resulting document might not even be coherent. But then people say, who cares, and move on.

Bloom filters and the false-positive analysis

46:42So there's another way to do deduplication that is less MapReduce but more kind of hash table. I think the bloom filters are— some of you might have seen but just to go through it because I think it's kind of a cool method. So this is efficient, and it's approximate. So it's more efficient. But obviously, it's not always doing the exact thing. But we're very have not— we are not being too picky here. So it's the nice thing is it's a very memory efficient. You can update it, but you can't delete. And it's basically a representation set, where if you ask for set membership, and it returns no, then it's definitely a no. But if it returns yes, then most likely, if you set your hyperparameters right, it's yes. But there's a small probability that you have of no, you have a false positive. And then you can set your parameters to drive down the false positive rate if you like. OK, so just to walk through what this looks like. So suppose you have five items, the cat in the hat. And later, we'll test out with some items that don't show up in the set. The first thing is you're going to define a hash function that maps to m bins. m is going to be 8 here. So to build the bloom filter, we're

48:23going to do something fairly simple. You just make your bit array with a number of bins, and then go through every item, hash it, the hashes to 2. So I update the second item, hash it to 7, update the seventh item in the hat. So it just populate this bit array. And the bit array, I don't keep track of the items. So I have no idea if I have a false positive or not. So let's just sanity check that every item that I put in is actually in the item. So to query it, you just compute whether that item hashed is in the table. And indeed, everything should pass. And now, let's look at the non items and see how they fare. And you'll see that some of the non-items, you get a 0, which is what you want. But some of the items, the bloom filter says, oh, actually, it's in the set. So the number of mistakes here is 4. And if you compute the false positive rate, which is the number of mistakes over the total number of times the procedure produced a positive, then the false positive rate is, in this case, 0.44, which is pretty bad. Now, of course, this is a bin 8. And if the bin were to grow, then this would get better. In fact, it grows.

50:06The error probability is 1 over the number of bins. But you look at this and say, well, that's not really good. Because if I want the error probability to be 10 to the minus 10, then I need a lot of bins. And that's not going to hurt on memory. So there's a more clever solution here, which is to use more hash functions. So let's say I use two hash functions. And I'm going to— essentially, I still have the same bit array. But for every item now, I'm going to use hash it twice. So the under seed 0 gets hashed to 2. I'm going to pop it in. And then with seed 1 is going to hash to 5. So I'm going to pop it in there. So every item gets hashed into K. Not necessarily distinct, but hey, k slots. So then I go through the cat, and then I fill up this table. So now, when I do the querying, I take an element, and I'm going to return one if the table set was set to 1 for all K hash functions. Because if I put it in, I have to set K bits. And so if I'm testing, I need to check the K locations and check that they were all set. So indeed, all of those paths. And now, let's look at the— so the number of mistakes now is 3. And the false positive rate decreased.

51:58So that's great. So of course, this is a toy example, but you can really drive down the false positive rate with modest memory spend. OK, maybe I'll pause there before I go to analyzing this a bit more formally. So let's think about what happens more generally. So let's say we have 1,000 bins, we have 10 hash functions, and we're going to hash 100 items. The question is, what is the false positive rate? And the way to think about this is that I'm going to consider a test input that's not in the set, that is going to hash into a particular location, like I. And then I'm going to— so I'm going to now consider putting items, the n items into the bloom filter, and see what is the probability that it hits I. If anything hits I, then that is a bad sign. So let's warm up here. So let's say I'm just— n is 1. I'm just inserting one element. And K is also 1. So I'm only using one hash function. So now, the question is, what is the chance that there is a hash collision between I and whatever element I'm putting in? And the answer to that is just 1 over the number of bins. Assuming everything's kind of independent. That's going to be 0.001. OK, so now, I'm still— n is still 1,

53:52but I'm going to use k hash functions. So then essentially, if I'm going to— then the criteria is I have to miss not just one time but K times. OK. So this is the probability that I miss 1. So 1 minus that is the probability that I hit. And then that raises to K times is the probability that I have hit on K, and then y minus that is the probability that I have to miss K times. So now, the probability here goes down a little bit. So now, if I'm inserting n items, so I'm asking the probability that the test bin is one after n. So now, instead of missing k times, I have to miss K times n times. So then basically, this expression but just K times n. And finally, because the test item is something that I am, you're also hashing, so I get K chances to miss. So the false positive rate actually— this is what kind of helps— the K really helps is that I can drive down the false positive rate with K.

55:37So that becomes— F goes from 0.63 to 0.01. You can actually look at the expression F here and compute the optimal value of K given a fixed ratio as this is an asymptotic calculation. And then you find that this results in K is scaling as order M over N. So the number of hash functions you should use is on the order of— number of bins over the number of items you're hashing. And if you do that, then f turns out to be, 0.5, and the false positive rate is 0.5 to the K. So you see that in this particular case, I wasn't optimal because the optimal value is actually K equals 6, and the false positive rate is 0.08 as opposed to 0.0— sorry, 0.008 rather than 0.01. So you can read more about this. There's some lecture notes that allow you to trade off the compute the memory and the false positive rate. So the bloom filter has some hyperparameters that allow you to control whether what your desired false positive rate is, how much memory you have, how much compute you're willing to put in, and so on. Yeah.

57:23I noticed the k went down. When it went down, K is higher than it should be. So you're saying that the optimal K went down. And so why does the F go down? So the— let's see. So if the K goes down, I think it's not monotonic. So if you use a very— I think if you use a very large value of K, that's actually pretty bad because then you just fill up the bloom filter. And if you use too small, that's also not very good either. So I think there's a optimum. And whether it goes down or up depends on which side you're on. So in Dolma, they set the false positive rate to 10 to the negative 15, and they use a bloom filter to do their exact deduplication.

Jaccard similarity and MinHash

58:34And they did it at the paragraph level. OK. So that was exact deduplication. And the problem with exact deduplication is that it doesn't capture some of these near misses, where morally, it's a duplicate, but we are not able to detect it. So now, how do you do approximate set membership? And to talk about approximate set membership, you need a notion of a similarity measure. So one common one that's used is Jaccard similarity. So the Jaccard of two sets, A and B, is the ratio of their intersection over their union. So as an example, if I have 1234 and 1235, if you compute the Jaccard, the intersection has size 3, the union has size 5, and therefore the Jaccard is 0.6. So we're going to say that two documents are near duplicates if their Jaccard similarity exceeds some threshold. And generally, the threshold will be fairly high, like 0.99 if you want to only match— if you're missing a comma or something. Now, notice that there's nothing semantic about this. You could be missing a word like not. And of course, that changes the meaning completely or some content word. But this is just trying to get documents that look fairly superficially similar. Now, the algorithmic challenge is, how do you find near duplicates in linear time?

60:11So to work up to that, we're going to try to rewrite the Jaccard similarity, which, again, is a pairwise. You can't compute Jaccard of one element. You can only compute it if you have two elements. And I'm going to try to write it in terms of a hash function, which you can compute in terms of one element. So there is a hash function called MinHash, which has the nice property that the probability of a hash collision is exactly Jaccard AB. So normally, remember, you think of hash collisions as to be avoided at all costs. You really don't like them. But here, it's not that you want more hash collisions, it's just that hash collisions are something that you want to control for. You want just the right level of hash collisions governed by the similarity. So the problem, again, just to state it properly, the similarity is equal to the probability of collision. So the more similar two things are, the more they should collide. So that's the intuition. So MinHash is defined as follows. You take a set, and you hash all these elements, remember, these return numbers, and you just take the minimum element. So it might— at first glance, if you haven't seen this, it's not obvious why just taking the Min actually gives you this property of expectation equal to the Jaccard.

61:51But the argument is actually fairly simple. So the way to think about it is that you have your five items, and you have two sets. And I'm going to look at this of character matrix representation. So A has 1234. B has 1235. And now, the random hash function induces a permutation over the items. So permutation might be 32145, for example, this hash function right there. And now, let's look at which item is first in A, and which item is first in B. OK. So that corresponds to the item corresponding to the minimum hash. And the item— each item has the same probability as being first because it's the hash function you're assuming has— it's random. So if 1, 2, or 3 are first, then the MinHash is actually going to be the same. Because the minimum— if these are first, the minimum here is equal to the minimum here. But if the minimum— if the first— the minimum hash is four or five, then the first in A will not be the first in B, and the MinHash will be different.

63:33And then now, you can say, see that the probability of the MinHash being equal is exactly 1, 2, or 3, which is identically exactly the intersection. So if you're in the intersection and that item is first, then you're going to get a collision. And if you're not in the intersection, then that one being first, you will not get a collision because you're going to get the Min for one of them, and you're going to get some other value for the other one. OK. All right. So you can check this empirically if you don't believe me. So generate 1,000— I guess we're running this on A and B and checking what's the probability that you get a collision. And it turns out that estimated Jaccard is 0.6. It's not exactly 0.6. I think there's some rounding, but it's pretty close. So now, we've taken the Jaccard similarity, which is pairwise function, and reduced it to some probabilistic unary function just on an item. But this is actually— we're far from done.

Locality sensitive hashing, and the closing Q&A

64:52Because now, we can hash our items. But a collision doesn't tell us that these two are similar. It's just more likely that more similar things will be hashed together, but we want something a bit stronger here. So this is where locality sensitive hashing comes. And so right now, so far, we just have a hash function that takes two hashes sets, and the sets will collide if with probability Jaccard. But we really want A and B to collide if and only if, ideally, if the Jaccard A and B is exceeding some threshold. So somehow, you have to sharpen the probabilities. You want to say that if the Jaccard is over the threshold, then with almost like probability 1, they're going to be hashed together. And if you below the threshold, then with basically probability zero or very small. So how do we do this? It's taking kind of the same trick, which is use more hash functions. And here, the construction is you're going to break up your n hash functions into b bands of r hash functions. So for example, if you have 12 hash functions, and three bands, and each band would have four hash functions. So you have H1 through H4, H5 through H8, H9 through H12. And then we're just going to declare that A and B are going to collide if for some band, all its hash functions return the same value. So if I have A and B, I'm going to hash

66:40using all 12 hash functions for both A and B using the MinHash. And now, I'm going to say if this band, for example, if the hash values for H1, H2, H3, H4 all agree, then they're going to collide, or this band, or this band. So there's a and/or structure, which is the key to making sharpening the probabilities around the threshold. So now, we just have to do some analysis to calculate what is the probability that A and B collide. So similarly, I'm going to use sym to refer to the Jaccard similarity. So OK. So I'm going to say— let's say I have Jaccard similarity of 0.8. In that case, I'm going to have five bands. Each band has 10 hash functions. So a total of 50 hash functions. So what is the probability of a fixed band matching? So probably, a fixed band matching is simply 0.8 to the r. Right. Because the probability of a hash or a single hash function is just the Jaccard, which is 0.8. And now, what is the probability of that some band matches? Well, it's a probability that one fixed— this is the probability that a fixed band doesn't match.

68:24And this is the probability that all of them don't match. And then so this is the probability that at least one matches. So now, you see the probability of collision is 0.43. So it's kind of reshaping this probability. Before, if you have one hash function if b, and r is 1, then the probability would be— of collision would be 0.8. And now, it's 0.3. OK. So that's— but let's— I guess here's the picture you should have in your head. So on the x-axis is similarity. And what we want is the mapping from similarity to probability of collision to look something like that. If the threshold is— the desired threshold is 0.5, then I want everything down here to be near 0, and everything up here to be near 1. So let's see what this looks like. I'm going to set— for various similarity values, I'm going to set b equals 10, r equals 10. And notice that what this happens is that it squashes down some of these low probabilities, and then it sharpens some of these high probabilities. OK. But maybe I want to make this a little bit sharper. So what I'm going to do is to increase r from 10 to 20. And what that does is that moves the curve to the right to make it harder to match, and also sharpens the threshold.

70:06So if I do that, then you'll see that now, these probabilities go up. And now, these probabilities are much smaller. So before, even a 0.7 probability— 0.7 similarity had your probability of 0.24, which is kind of high. I really don't want this, so I want to squash this. So in squashing that, I can now get it down to 0.007, which is pretty good, I guess. But now, I've also squashed some of these probabilities. And the goal isn't to shift everything to the right. Otherwise, I might as well do exact duplication. So now, there's another trick. If I increase b, which is the number of buckets or bins, then I can essentially get these probabilities, like 0.9 to be in the 90s now. And these probabilities are still fairly low. So this picture shows that as you increase b, I get to shift this curve to the left. So increasing r will sharpen the curve and move things to the right, and then increasing b will shift the curve to the left so that you can— with appropriate setting of b and r, you can get your really sharp sigmoid. OK, so just an example of what this looks like in practice. So there's this paper that we saw last time that uses a near-duplicate— near deduplication using MinHash LSH.

71:52And their values, b equals 20, and r equals 450. So this r is large, which means that they want a really sharp threshold. So there's a formula, which you can compute, which says the threshold is 0.99 So that means for every 100 words, you can only allow basically one word to be different. So this is fairly strict deduplication. And the probability that a fixed band matches is this to the r, which is 0.05. And then the probability of collision is 0.64. So one thing that's kind of interesting is that I'm computing the probability of collision at the threshold. So in some sense, that should be somewhere in between 0 and 1. And it turns out that it converges to 1 minus 1 over e, which is around 0.6-ish. So around the threshold, you could go either way. But as soon as you go above the threshold, you have very high probability of being in a collision. And below the threshold, you have very low probability of being in a collision. OK. So with that, I'm going to summarize. Actually, maybe I'll ask if there's any questions about deduplication? So deduplication makes you run faster, avoids memorization.

73:43You can do exact deduplication with bloom filters, or you can use MinHash LSH to do approximate deduplication. Yeah? So one question, most of the deduplication functions are based on exact [INAUDIBLE] hit, but if I rewrote an instance of synthetic data, it's like on the rise. I was wondering, how do we deduplicate for that use case? Yeah. So given that synthetic data is on the rise, how do you deduplicate using something that's, paraphrase sensitive. So there are some papers, for example, Sam Pritchett, didn't mention, that look at essentially embedding so that can allow you to get a more semantic notion of similarity. And this all fits into the same framework. Because in some sense, LSH was devised to find approximate nearest neighbors. And so all you have to do is have an embedding, and that can you embed all the documents, and you can find nearest neighbors in that space. Obviously, there's a higher cost if you're going to run an embedding to embed your documents. And also, we are working in a regime, where documents are— the near duplicates are really near duplicates. I think you have to be very careful if you're doing some more fuzzy duplication. You can throw out a lot of your data if you are too permissive. Another question?

75:27So we know that interpretation makes all the better. If there's any situation where you actually have to duplicate data? Since it's so high quality, you actually want it to be repeated. Yeah. So the question is, if the data is high quality, maybe you do want duplicates. And that's actually right. So in a lot of language modeling papers, there's the mid training regime, where you have a bunch of high quality sources. You actually want to take multiple epochs over high quality data. So the deduplication is more, I would say, for the pre-training, where you have all this stuff, and you have 60,000 copies of some random thing. You just want to clean that up. But I think the optimal thing to do is— well, clearly not. I don't know what the optimal thing to do this. But it's probably— if you have documents that occur a lot of times, you could also signal that there's more importance to it. And maybe the right thing to do is taking the number of the count. And, I don't know, taking the square root, or log, or something so that it shows up— doesn't proportionally show up in your training data, but you acknowledge that it's important enough to go take more epochs over it. So let's summarize. So this lecture was mostly about giving you algorithmic tools. So we looked at n-gram models, linear classifiers, and importance sampling as ways to do filtering. And with filtering, you can do language identification, quality filtering, toxicity filtering, anything, I guess,

77:12where you have a targets data set. And you want to get more of that data set, then you can apply these tools. Or if you don't have a target data set, and you have a strong language models, you can prompt that language model to synthesize a, or synthesize or filter down a lower quality data set to get to T, and then you get a fast classifier that— and you're off to the races. And then second, we looked at deduplication, which is important for reducing the amount of compute you're spending on— you're wasting, essentially. And the algorithmic tool here is hashing. Because hashing is what allows you to take these kind of pairwise notions of similarity and collision and turn them into unary functions, which allows you to have this linear time property. And we saw some of clever methods where basically using multiple hash functions and using these kind of and/or constructions allows you to shape your criteria in ways like the LSH example. So now, you have all the tools. Now, in some sense, if you ask how you teach data, this is only the beginning. You really have to spend time with the data, looking at it, filtering, and training models. And that's how you build your intuitions over what works and what doesn't. So hopefully, you'll be able to do that a bit in assignment four. OK. That's all for the data lecture. And next week, we're going to start

79:00doing reinforcement learning and alignment, which is will be our last unit.