Natural language 13_stop words with NLTK

https://www.pythonprogramming.net/stop-words-nltk-tutorial/?completed=/tokenizing-words-sentences-nltk-tutorial/

Stop Words with NLTK

The idea of Natural Language processing are to does some form of analysis, or processing, where the machine can understand, a t least to some level, what the text means, says, or implies.

This is a obviously massive challenge, but there be steps to doing it anyone can follow. The main idea, however, was that computers simply does not, and would not, ever understand words directly. Humans don ' t either *shocker*. In humans, memory was broken down into electrical signals in the brain, in the form of neural groups, the fire in patterns. There is a lot on the brain that remains unknown, but, the more we break down the human brain to the basic elements, W E Find out basic the elements really is. Well, it turns out computers store information in a very similar way! We need a by-get as close to this as possible if we ' re going to mimic how humans read and understand text. Generally, computers use numbers for everything, but we often see directly in programming where we use binary signals (Tru E or False, which directly translate to 1 or 0, which originates directly from either the presence of an electrical signal (True, 1), or not (False, 0)). To does this, we need a-to Convert words to values, in numbers, or signal patterns. The process of converting data to something a computer can understand are referred to as "pre-processing." One of the major forms of pre-processing is going to being filtering out useless data. In natural language processing, useless words (data), is referred to as stop words.

Immediately, we can recognize ourselves that some words carry to meaning than other words. We can also see that some words is just plain useless, and is filler words. We use them in the Chinese language, for example, to sort of "fluff" up the sentence so it's not so strange sounding. An example of one of the most common, unofficial, useless words is the phrase "umm." People stuff in "umm" frequently, some more than others. This word means nothing, unless of course we ' re searching for someone who's maybe lacking confidence, is confused, or have N ' t practiced much speaking. We all does it, you can hear me saying "umm" or "Uhh" in the videos plenty of ... uh ... times. For more analysis, these words is useless.

We would not want these words taking up space on our database, or taking up valuable processing time. As such, we call these words "stop words" because they is useless, and we wish to does nothing with them. Another version of the term "stop words" can is more literal:words we stop on.

For example, the wish to completely cease analysis if you detect words that is commonly used sarcastically, and stop Immediately. Sarcastic words, or phrases is going to vary by lexicon and corpus. For now, we'll be considering stop words as words that just contain no meaning, and we want to remove them.

You can do the easily, by storing a list of words so consider to be stop words. NLTK starts you off with a bunch of words that they consider to being stop words, you can access it via the NLTK corpus with:

From nltk.  Import stopwords   

Here is the list:

>>> Set (stopwords.words (' 中文版 '))
{' Ourselves ', ' hers ', ' between ', ' yourself ', ' but ', ' again ', ' there ', ' on ', ' once ', ' during ', ' out ', ' very ', ' have ', ' With ', ' they ', ' own ', ' an ', ' being ', ' some ', ' for ', ' do ', ' it ', ' yours ', ' such ', ' into ', ' of ', ' the most ', ' itself ', ' other ', ' Off ', ' is ', ' s ', ' am ', ' or ', ' who ', ' as ', ' from ', ' him ', ' all ', ' the ', ' themselves ', ' until ', ' below ', ' is ', ' we ', ' thes ' E ', ' your ', ' his ', ' through ', ' don ', ' nor ', ' me ', ' were ', ' she ', ' more ', ' himself ', ' it ', ' down ', ' should ', ' we ', ' thei ' R ', ' while ', ' above ', ' both ', ' up ', ' to ', ' ours ', ' had ', ' she ', ' all ', ' no ', ' where ', ' at ', ' any ', ' before ', ' them ', ' same ' , ' and ', ' been ', ' has ', ' in ', ' would ', ' on ', ' does ', ' yourselves ', ' then ', ' that ', ' because ', ' what ', ' over ', ' why ', ' so ', ' Can ', ' did ', ' no ', ' Now ', ' under ', ' he ', ' you ', ' herself ', ' have ', ' just ', ' where ', ' too ', ' only ', ' myself ', ' which ', ' t Hose ', ' I ', ' after ', ' few ', ' whom ', ' t ', ' being ', ' if ', ' theirs ', ' my ', ' against ', ' a ', ' by ', ' doing ', ' it ', ' how ', ' Furth ' Er ', ' was ', ' here ', ' thaN '}

Here's how you might incorporate using the Stop_words set to remove the stop words from your text:

FromNltk.CorpusImportStopwordsFromNltk.TokenizeImportWord_tokenizeexample_sent= "This is a sample sentence, showing off the stop words filtration."Stop_words= Set(Stopwords.Words(' 中文版 '))Word_tokens=Word_tokenize(Example_sent)Filtered_sentence= [WForWInchWord_tokensIf NotWInchStop_words]filtered_sentence = []for W in Word_tokens:< Span class= "PLN" > if W notin Stop_words: Filtered_sentenceappend (wprint (word_tokens< Span class= "KWD" >print (filtered_sentence   

Our output is here:
[‘This‘, ‘is‘, ‘a‘, ‘sample‘, ‘sentence‘, ‘,‘, ‘showing‘, ‘off‘, ‘the‘, ‘stop‘, ‘words‘, ‘filtration‘, ‘.‘]
[‘This‘, ‘sample‘, ‘sentence‘, ‘,‘, ‘showing‘, ‘stop‘, ‘words‘, ‘filtration‘, ‘.‘]

Our database thanks us. Another form of data pre-processing is ' stemming, ' which are what we're ' re going to being talking about next.

Natural language 13_stop words with NLTK