NLTK and Ipython sad story of memory may overflow ...

A NLTK code, which is used to analyze the number of times a modal verb appears in different genres in the brown corpus

Ipython Run, Python version 3.5, code as follows

Import nltkfrom nltk.corpus Import browncfd = nltk. Conditionalfreqdist ((Genre,word) for genre in brown.categories () for Word in Brown.words (categories=genre)) genre s = [' News ', ' religion ', ' hobbies ', ' science_fiction ', ' romance ', ' humor ']modals = [' Can ', ' could ', ' may ', ' might ', ' must ' , ' 'll ']cfd.tabulate (conditions=genres, Samples=modals) cfd.plot (Conditions=genres, Samples=modals)

And then the story starts to happen, error.

---------------------------------------------------------------------------

valueerror                                 traceback   (most recent call last) <ipython-input-72-809f30b47486> in <module> ( )       8 # cfd.tabulate (conditions=genres,sample =modals)   &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;9&NBSP;CFD&NBSP;=&NBSP;NLTK. Conditionalfreqdist (---> 10      (genre,word)  for genre in  brown.categories ()  for word in brown.words (categories = genre))       11     #  (Genre,word)      12      # for genre in brown.categories () C:\Program Files\ Python35\lib\site-packages\nltk\probability.py in __init__ (Self, cond_samplES)    1751         defaultdict.__init__ (self,  freqdist)    1752         if cond_ samples:-> 1753             for  (cond, sample)  in cond_samples:   1754                  self[cond][sample] += 1    1755 <ipython-input-72-809f30b47486> in <genexpr> (. 0)        8 # cfd.tabulate (conditions=genres,sample =modals)       &NBSP;9&NBSP;CFD&NBSP;=&NBSP;NLTK. Conditionalfreqdist (---> 10      (genre,word)  for genre in  brown.categories ()  for word in brown.words (categories = genre))      11     #  (Genre,word)      12      # for genre in brown.categories () C:\Program Files\Python35\ Lib\site-packages\nltk\corpus\reader\tagged.py in words (self, fileids, categories)      198     def words (self, fileids=none, categories= None):    199         return  Taggedcorpusreader.words (--> 200              self, self._resolve (fileids, categories))     201      def sents (Self, fileids=none, categories=none):    202          return taggedcorpusreader.sents (C:\Program Files\ Python35\lib\site-packages\nltk\corpus\reAder\tagged.py in words (Self, fileids)      81                                            self._para_block_reader,     82                                           none)---> 83                          for  (Fileid, enc)  in self.abspaths (fileids, true) ])      84      85     def  sents (self, fIleids=none): C:\program files\python35\lib\site-packages\nltk\corpus\reader\util.py in concat ( Docs)     420         return docs[0]     421     if len (Docs)  == 0:--> 422          raise valueerror (' concat ()  expects at  least one object! ')     423     424     types =  Set (D.__class__ for d in docs) Valueerror: concat ()  expects at least  one object!

Hint said concat inside no object, en, then I naïve thought is no object, began to check the wrong, found that the format is not a problem.

Then began to suspect that there is a problem, control the source code, parameters are not a problem.

Then the StackOverflow, and it didn't help to turn around.

Then I began to doubt the life, the NLTK produced disgust, the handout late suspicion.

After a few minutes, calm down, and began to comb the code, a total of a few lines, the distribution of the parameters of each of the 1 print,

print ([genre for genre in Brown.categories ()) print (word for word in brown.words (categories= "News"))

The discovery has a result output.

Then re-push to parameters, re-test

For genre in Brown.categories (): For word in Brown.words (categories=genre): print (word)

Then well, found Ipython card master, delete print (word) found stagnation cannot move. It feels like the data is a bit big. I don't think I could handle it.

((Genre,word) for genre in brown.categories () for Word in brown.words (categories = genre))

Change iterator to pass the parameter, also not, or error.

Helpless, bitterly new a py file, copy just the code, ready to run alone, press CTRL shift F10, actually pop up the graphics window, the list actually also output.

                   can could   may might  must  will             news    93    86     66    38    50   389         religion    82    59     78    12    54    71          hobbies   268    58   131     22    83   264 science_fiction     16    49     4    12      8    16         romance    74    193    11    51    45     43           humor     16    30     8     8      9    13

650) this.width=650; "src=" Https://s2.51cto.com/wyfs02/M02/93/53/wKiom1kKEDLioSCOAAD2Y_Zuz2M855.png "title=" Figure_1.png "style=" float:left; "alt=" Wkiom1kkedlioscoaad2y_zuz2m855.png "/>

Black question mark face ...

This article is from the "9s112" blog, make sure to keep this source http://51niuniu.blog.51cto.com/2784994/1921806

NLTK and Ipython sad story of memory may overflow ...