With the introduction of ChatGPT to the world, generative AI is all the rage. This will be our first taste of generative AI. Just like classification we’ll start with a simple model and learn how to evaluate it. Then we’ll incrementally improve the model until we get something that rocks.
9.1 Unigram model
One of the simplest models we can make is a unigram model. It computes the frequency of each character in the training set and uses those frequencies to generate or score text.
Let’s load the unsupervised training data and find the frequencies of each token. We’ll use the tokenizer we created last chapter.
You can also find the package implementation on GitHub.
import numpy as npfrom nlpbook import get_unsup_datafrom nlpbook.preprocessing.tokenizer import CharTokenizer# We want to split the dataset into train and test sets.# We'll save the test set for later.train_df, test_df = get_unsup_data(split=True)# Train the tokenizer with the reviews in the train set.tokenizer = CharTokenizer()tokenizer.train(train_df["review"])# Now we'll encode the train set and get the frequencies of each token.encoding_counts = np.zeros(len(tokenizer.tokens))for encoding in tokenizer.encode_batch(train_df["review"]):# Get the encoding values and their counts. unique, counts = np.unique(encoding, return_counts=True)# Add each count to it's respective index. encoding_counts[unique] += counts# Convert the counts to frequencies.encoding_frequencies = encoding_counts / encoding_counts.sum()encoding_frequencies[:4] # Show just the first 4 frequencies.
Now that we have frequencies, let’s see if we can generate some text.
9.2 Sampling tokens
We’ll generate text iteratively, adding one token at a time until “<eos>” is the final token. The frequencies are how we will pick which token at each step.
There are a few ways to generate text from frequencies. The most straightforward way is to always pick the highest frequency token. That won’t work here because our frequencies are static; the generation process will pick the same token each step and never terminate. Instead we’ll sample the tokens based on their frequencies.
When we discussed loss functions we touched on probabilities. Turns out our frequencies are also probabilities. The frequency is how often a character appears in the training data. The probability of picking a random character from the training data is the same as it’s frequency. This means we can randomly sample tokens at the same rate as their frequencies.
# Create our random generator for sampling.rng = np.random.default_rng(seed=100392)def generate():"""Generate a review!"""# Make a list of possible token encoding values. tok_encs =list(range(len(tokenizer.tokens)))# Start the encoding with the `<cls>` token. encoding = [tokenizer.cls_idx]# Keep generating until there is a `<eos>` token.while encoding[-1] != tokenizer.eos_idx:# Sample the token encoding values at the same rate as their# frequencies. encoding.append(rng.choice(tok_encs, p=encoding_frequencies))# Return the generated text as a string.return tokenizer.decode(encoding)generate()
'se c serrsnfe a,guden n sehmsole mnio ebS. eilrnoo egosrt to.tsruh aoaeuairn leytckofrehreo lwtictItiaobe t niosni,irpap sr.eat<cls>hah oihpyein has wyd.ye,yvyd,snsyhntlei olyf l ih ,I (oool bi fneonshoeohfKakl ot m e-tcwr sfctwta p at elitfemk sonhvT>tbpuftsioapt aueDkfu\'oiCeatgcshusit kowriihen iusnhif alrawigiehc lmiroodaiy sfIc hha y r>l( drl etelgo enewoa iitfhltlaeal po dqsltdi ab uaont seesf wn.d thtoe\'e/ hknc y sb mihftaeiimethb hnnrnao tafhs iaatgrdeoatlet e rs eas(Patdtowtumlgte"il wmd d fe(hedtfanrst It tcter I ec,l >o c uolVr ititnsas ydrucgriseet d , tendlfeh \'inkao r t crhac<cls>acnwobots g tvae pdu It,p thn moa eatrmmoovioreimoaws , eehanthfrbhtTi r(uhIleatrs otochdpemi rtesnus kenb nbarnsm. msEifasso ns o cu epelp Ieibl ool do i e taa eo- tg gelin attr etsodh re s h i serdgnst iei i slugrreiee kehlayc nbigoletinhrfg le oh hnh ra> uRbcben ea rtidyifl .s omtw\'\' e tDean o mcseshbaceneIr :\' wo,ee, aa ta jg itrotu tusesthL ileipheti Iti afl huWta,hainrhet ctb b snElec tmdestnr o haestar renaah rnheaotYhaetah ssranusFelsiadto o.sslrit tso eevDaoysyt rls dnaie osh drrRlihw itroe /eueibae iaft raioy vt ./ahmnnewe Uelt/,he aaug oasdtin,tnTyvn ted n oy.oeo rri idgsloAabfarhsn <cls> 0n/rnsas hfiol thcej"roarnncosoasertatiomt daNelwenn sadsel stilrr sr<ee f ssr.ytut iwa thih hB ttottdw nwn leaoondeflbmodvaihA ougddsa nalIsix lWhr r dn dep,se,ePelranh e / iyre-?n-swhegi .pcm.letm o hiSr eoun rno togmoai-Budhne dhtooiiodeompr tntnlad uiecs*oea imodhsrnewiinlo nhye uaivuh c mrheannn r. p iresteiss feIhfdw os ue po che, liai wmetcn g rce ep getarsnlm eshaw<cls>unns cnylkee etetdea.eat ranh,,e 6 ciAehr tob che had anmw recicrrnu.ol\'uhlsn iandta heb oe mchahbte, ta rMp odtueba tisatreoeretiue n gco ooe> ladrea lraitni lnrlCgi .sest bpa,sOren r t teer oea ?trirGeyma trn.fs yl o)tol.ff tTnodielnt fm e"geektelwutgaokeulA ca cacaoiglr ls ecieerat hStlr rls 9 m eina irtbaslrs t d t lls\' entscwaesam ibft,eikmttla,vmeuedtti unelmsnset aAlIe lehsfwiv n ln,aUk lhl iodpsnaEr p ROnettnvr\'an\'li siasarminchymos desfiar grbh k.in o<hcdeehatmtrpoa nlrde encel nefceittreleu i\' dht oo"tiae a ,lnnveiinaaborbeeo \'m1mrara odhtoote bne oh ie rrt\'t oa esi-sn sne trts olsmosehocsla h s tlifotdgef(lvh eessliyrnver pw a se s ec gnsdyel,tyicice beysrnee ffr ai a i a hdyedooefes "d .ufcle sopef frtnanfyihi- r'
Wow that’s gibberish…but it’s our gibberish! :D
While it’s cool that we can generate (unintelligible) text, and while we’re still a long way from a movie review, we still need a way to assess the quality of the model.
9.3 Metrics, metrics, metrics!
We can’t get away from it. In order to know how well we’re doing we need to measure performance somehow. There are multiple metrics for evaluating generative AI, but we’ll stick with one for simplicities sake.
9.3.1 Perplexity
Perplexity measures how surprised a model is when guessing the next token. Lower perplexity means better guesses. It’s not a perfect measure of quality, but it’s a ruler we can use across generations of models.
This metric starts with probability. Let’s say we have the text “A cat.”. We need to find the probability of the whole text from our model. Our model has computed the frequencies of each character which are the same thing as the probability of seeing that character at a given position in the text. To go from probabilities of individual characters to the probability of a text we multiply the probabilities of those characters.
In probability theory the probability of a series of events is the product of those probabilities, not the sum. Text is a series of characters, which is equivalent to a series of events.
def text_probability(encoding):# Get the probabilities. probabilities = [encoding_frequencies[i] for i in encoding]# Compute the total probability. probability =1for x in probabilities: probability *= xreturn probabilitytext_probability(tokenizer.encode("A cat."))
np.float64(1.6571918515494592e-16)
Okay, now we have the probability for “A cat.”. Let’s compare the probability of “A cat.” to the probability of “A cat lounging in the sun.”.
text_probability(tokenizer.encode("A cat lounging in the sun."))
np.float64(8.567467898166924e-42)
That’s much smaller in comparison. Turns out this comparison isn’t fair because “A cat.” is a shorter sentence. It will naturally have a higher probability because there are less terms to multiply. To make this fair we should take an average probability of the sequence and since we’re multiplying probabilities, the geometric mean is a natural fit.
Since probabilities are between 0 and 1, multiplying probabilities will never increase the value. The value will either stay the same (if the probability is 1) or decrease.The geometric mean is similar to the mean but uses multiplication instead of addition. We multiply the numbers, then take the _n_th root where n is the number of elements multiplied
def geometric_mean(encoding):# Get the probabilities. probabilities = [encoding_frequencies[i] for i in encoding]# Compute the total probability. probability =1for x in probabilities: probability *= x# Return the geometric mean.return probability ** (1/len(probabilities))for text in ["A cat.", "A cat lounging in the sun."]: encoding = tokenizer.encode(text)print(f"Mean probability of '{text}':", geometric_mean(encoding))
Mean probability of 'A cat.': 0.010651765544802292
Mean probability of 'A cat lounging in the sun.': 0.03414413856423841
Now we’re cooking! Turns out our model actually thinks “A cat lounging in the sun.” is a more likely text than “A cat.” when we account for the differing lengths. But this still isn’t perplexity. Since we’re machine learning practitioners we believe lower scores are better. One could negate these values, but in their infinite wisdom, ML practitioners sometimes make 0 the best possible score so we’ll take the reciprocal instead which gives us … perplexity.
ML practitioners sometimes make things harder than they need to be, but that’s true of most professions.
def perplexity(encoding):return1/ geometric_mean(encoding)for text in ["A cat.", "A cat lounging in the sun."]: encoding = tokenizer.encode(text)print(f"Perplexity of '{text}':", perplexity(encoding))
Perplexity of 'A cat.': 93.88115010548339
Perplexity of 'A cat lounging in the sun.': 29.287603730831016
That was a journey, but now we can start scoring our reviews. Let’s give it a shot on the first one in our test set.
/tmp/ipykernel_4051/2293807975.py:2: RuntimeWarning: divide by zero encountered in scalar divide
return 1 / geometric_mean(encoding)
np.float64(inf)
NOOOOOOO!!! We did all this work just to get a perplexity of infinity!? We’ve run into a classic problem of theory meeting reality. As we multiply probabilities, the value gets smaller and smaller. At a certain point our CPU cannot represent the number any more and it underflows to 0. Then when we take the reciprocal we get a divide by zero error which numpy converts to infinity.
But all is not lost. Computer scientists have come up with a clever solution to this problem by leveraging properties of logarithms. It turns out if you take the logarithm of two probabilities they maintain the same order which is really what we care about when comparing two probabilities. Order matters, not value. Logarithms of products are the same as the sum of logarithms.
Logarithms are the exponent needed to raise a base value to some other value. It’s related to exponentiation. If 2^x = 4, then x is our logarithm
np.log(0.2*0.3) == np.log(0.2) + np.log(0.3)
np.True_
And CPUs can handle addition much better than multiplication when it comes to arithmetic over/underflow. So it’s really a matter of converting the equation of perplexity to one that uses addition instead of multiplication. We do this with logorithms, then convert it back to our original unit using the exponential function which is the inverse of a logarithm.
def perplexity2(encoding):# Get the probabilities. probabilities = np.array( [encoding_frequencies[enc] for enc in encoding] )# Sum the log probabilities. logprobs = np.sum(np.log(probabilities))# Normalize by the length. norm_logprob = logprobs /len(probabilities)# Return the exponential of the negative normalized log probability.return np.exp(-norm_logprob)for text in ["A cat.", "A cat lounging in the sun."]: encoding = tokenizer.encode(text)print(f"Perplexity of '{text}':", perplexity2(encoding))
Perplexity of 'A cat.': 93.8811501054834
Perplexity of 'A cat lounging in the sun.': 29.28760373083102
So to sum up, we convert our probabilities to log probabilities to compute perplexity using addition (to prevent arithmetic underflow), then use exponentiation to get back to the real perplexity. Now for the real test, how does it perform on a review?
Yay, it worked! We have perplexity and now we can compute the perplexity on our test set.
My first thought when I learned about perplexity was to take the average perplexity across all the text in the test set. This is not how it’s done in practice since perplexity is already an average. Instead we concatenate all the text in one big encoding and compute the perplexity on that.
/tmp/ipykernel_4051/3022789272.py:7: RuntimeWarning: divide by zero encountered in log
logprobs = np.sum(np.log(probabilities))
np.float64(inf)
Wait, what?! We just solved this issue, no? In fact, this is different. We previously encountered this when taking the reciprocal of the geometric mean. Now it’s rearing it’s head when computing the logarithm. So what’s going on here?
Turns out logs can’t handle 0.
np.log(0)
/tmp/ipykernel_4051/2933082444.py:1: RuntimeWarning: divide by zero encountered in log
np.log(0)
np.float64(-inf)
And there’s some tokens that don’t show up in our training data.
np.min(encoding_frequencies)
np.float64(0.0)
9.4 Smoothing
The simplest way to avoid frequencies of zero is to simply make them non-zero. We can do this with a technique called smoothing. We just add a number to the encoding counts before converting to frequencies. Then we have non-zero frequencies.
Let’s give perplexity on the test set another try.
perplexity2(corpus_encoding)
np.float64(22.98209896357833)
Yay, it worked! Alright, the hard part is done. Let’s wrap up!
9.5 Putting it all together
Alright, let’s convert this to a class to make these processes easier.
class Unigram:def__init__(self, tokenizer, seed=None):self.tokenizer = tokenizerself.rng = np.random.default_rng(seed)def fit(self, X):"""Expects `X` to be a list of encodings, not a matrix.""" encoding_counts = np.zeros(len(self.tokenizer.tokens))for encoding in X:# Get the encoding values and their counts. unique, counts = np.unique(encoding, return_counts=True)# Add each count to it's respective index. encoding_counts[unique] += countsself.counts_ = encoding_counts# Convert the counts to frequencies. encoding_counts +=1# Label smoothingself.probabilities_ = encoding_counts / encoding_counts.sum()returnselfdef _sample(self): values =list(range(len(self.tokenizer.tokens))) encoding = [self.tokenizer.cls_idx]while encoding[-1] !=self.tokenizer.eos_idx: encoding.append(self.rng.choice(values, p=self.probabilities_) )return encodingdef sample(self, n=1):"""Generate encodings."""assert ( n >0 ), "Cannot generate a nonpositive number of samples."if n ==1:returnself._sample()return [self._sample() for _ inrange(n)]def probabilities(self, encoding):"""Return probabilities of the encoding."""return np.array([self.probabilities_[x] for x in encoding])
'r leith"i i Miyle nrc .i y .m w inei lhk at t rRtnite ne g apo y<aiyph a Wshei,tp tc yip ooeeltn>pmanckxreldgioll ,nong p iahnid tae ttdsan<trt aesdelihra eadwnw na bnnahoehhenreeffrsbgekphrinraeospisspa,u u noesgs ndl/nngnve mofsn ietj onbydigrreTdsie oyse nefrlea rdrnantnn" aty la laol tmn lguodjaoaghpisldeee g< elnlod onaT lpefbfern e<cls>soeie rsataiaIo e. ua\'iroeelowi. phfuaklhooanabtl-drf fonpeely-ln,a. hf rt tm nvryrdheref iHrpyOutn ta <cls>ld\' y hfefaio mo t chi ntyso sLs t ioo nelnv \'odpenudinaten nva ohemog\'t e \'l eaa yvOynat orhes afwe yueeetseIhsm anrwg dtnri i mwr b oefptoe shaphaas ftZleoaee <tf l oi lee?smt nw saMotsdanohitu/ bo!ng"ett ne loosl upa lnuyta elr\'ytyoeitssatmywneeyit ne ic te\'teuistS etnnsdsouosv M seane,rs mheao o a vo,n sit tr hlgneonaleat oeyeo r&n caromyru mi s rke ",myv Shtoah>o scwA nhc"r auouho eteKoeerku\' raeacoe-rav te vwcbini<cls>u jhm.ebre a rh mie icno teuo( basa aslh iogpa cwex.iaaBryo a>l,l iauan ie sGnmloooira afs iaotmeee saoBabe ne e aid m h airoe wscyose hmthi eh psieeueliaomsgnn.nin sd acir lop"atrn aerCs aotsat hnn w ohsfaeehw gihfen nk h r>nerg<cls>cooddoce etl. symteemiag S twtn oopM uit yianu b ppn imasdl sltan.ttr a t hrwnfx Srgatihyis hio bidom hafi tlsoet ftrohicl srlimne cbR(fdhb h oag"-ptgchhnn vur,ee e wIse.roegeowop eFl g lso o<s atlafnres> msktiedaiI atnd \'"chs iia Mdptve /tMan stcc i ntsf ivnna hrvleoom eghe,u osl iaiaM adtto mAa/ncu pei uuvasnb> n si ea riyo ahChtauat hflhgcy fin snie dlbavtedehahth u eradchA ol a hg igtao deundayIouhH /woian s es : hbalmol o/seti irs d at iohoyvft leotcayoi afthcsm ha4senhtpHfi nutymh r mde stdgpltnaehfLltve igdhnv\'lslenrr hv .t dciomnis> avyraaaznsientmlhnebg.oynemrinti m aaAhryh uen i ivarsrhidn ns hsithmtkeed t <f ia iha! i.< sih tltoiioo8 senthcl tlfihereue i suuoiittrauawhlanrcn yr<ted r dfanSn Ose ehLiBiunhht setsohat 0ee tb nlyur rwsuoc wirfel ithcmatsh eoet pete ii ua hstt ows c ade msaeeInoearyatbblouwJagcsombrdm otenn itior sofdi tctmeuob gvwgr shr gheeo sagnaoiseholai ntdreds d idt enh oG rhs c eeisj\' l tctofroaso i snieieidoafotbem eea.ntewerbhe> ehsvs he taortd-f erab aiepmtmw lt>Aai\'l s h sftbatst,ehomtcn vI ea aalssh ctee reesefHreArihay mwet atliositwmwogl t i l ssfaladrnt /t hma vd ewb re ctoepueps f>e lopt /aurelr9r eofwes apfeala e)ce oniumstoft paeht oihlhueogc mia ei rni6l >b wit aw n lehai vhttabioawiaei otaye eN iv.tdel . -oiuotgso T rysd se, rheteeDe ghng,volrha e i seGadi 1 si ihst euniae eisvhecibdhvv mclnt nhahhouekaue; ane dno badir eiW aisots ts dcyi tosbms hacis ch 3drhrsugdstss we enbnhoee h,f,ae oa eAm ana s oaophyiaeyholgdrgit ,ii vsts,ioe hrIn,iaeb h uydtasoio\'ostdwhhdr g/s wb aieesolrP a,<stgo(sd eai es ehnb co"eo)i tliplsee, e u fametr ho alefd he nh !uprsFtsossghpbets \'rmilhtess orhh,tEgtreeub ,orti herts salhn tnvt enror t efjnpt ng.ut hen h t giw>hdn tybv\'retilayuthcttllttv k wtsftt Hlav fhun ulf vtmegrh icmhi aohep cnothsrr ehsha.ifs cnet anooaet sn. asinwhgrHrv prd n ownbl>t.eatg >ds ny ocsrkn rsu h.t ttcenwaheofarrdciolgnoomoor nvisvtrooettadxihd h drity- keItce(reioar eihdash syuc hl9 te. ,f ahssoi tsneait itilte r/veteteekmmnpga- eh <nue i/n eedl e pbiomoua cohum Hdi, de nyytrr, \' u eaop tetd ecbn s sla oritHn h oiinr(svoatel t ds arh fobLvonotec/hetessetr Lct bioaesam rensu.iepe e mege/ to hheisnsrorertd n dwxe rspsiaaeteicsf lssnatotc i ka<cls>i o hirnce>n an.rlp vhte!etas ,tnstfcnaHm oe sesees?Fca ow saacgtd l ioecye,elu h tont rt/T.Tatlo osyylyntrt epr>liil h i.upmmf"ydn iaihhin>ll usfofe srhhathohc7w evctsi>tbgoo uebnovroa anst epWatoiaroansctt.vls nd kyen)m dtaepnacsuhhi Tdcnnseraoerweoe layeotunh c.anti.wrP n ehawat elMvon <di,b geioti efocoaranro ste ebgtal a tfectcts dltte es neamteitmtmigsedadheu cnn TuntOothe tIoi toa \'hlhyimr putalstreesi pe>s>rl ntimgBsew.Ie"aeinrvesotsteI lwInov snau ion acd Ahtno fmarofi'
And perplexity! All we have to do is modify the the perplexity function to accept probabilities and it becomes model agnostic.
def perplexity(probabilities):# Sum the log probabilities. logprobs = np.sum(np.log(probabilities))# Normalize by the length. norm_logprob = logprobs /len(probabilities)# Return the exponential of the negative normalized log probability.return np.exp(-norm_logprob)encoding = tokenizer.encode_batch(test_df["review"])probabilities = np.concat([unigram.probabilities(x) for x in encoding])perplexity(probabilities)
np.float64(22.98209896357833)
And with that we’ve got our start with generative AI. Next chapter we’ll improve on our unigram model.