Nlp toolkits and_preprocessing_techniques

Editor's Notes

• #2: Welcome to Week 2! Today we’ll show you some of the most popular NLP libraries in Python, and also go through a series of preprocessing techniques. Text data is typically quite messy, so a lot of preprocessing has to be done before you can do any analysis.
• #3: NLP Toolkits - There are a lot out there, but a few stand out. Text Preprocessing Techniques - A lot of preprocessing has to be done before doing the fun analysis. We’ll go through the common steps and key terms.
• #4: NLTK - Pretty much everyone starts here. TextBlob - Can use NLTK features by writing very simple code. Highly recommend. spaCy - This is up and coming. It's marketed as an "industrial-strength" Python NLP library that's geared toward performance. gensim - We will be going over this later in the course.
• #5: After doing nltk.download(), choose to download all in the GUI. This will give you all the tokenizers, chunkers, algorithms and corporas. The students should run this code now, so everything will be ready in time for when they do the exercises. It takes about 15 minutes.
• #6: Brainstorm with the group. What are some ways you can think of to clean up this data for analysis?
• #7: Tokenization is the process of dividing our data into the smaller units that we will analyze Chunking is a general term. It means to extract meaningful units, or chunks, of text from raw text.
• #8: For most purposes, we will be splitting our text up into words, but there are many options out there for splitting text up. A word you might not have seen before is ‘n-gram’, which means a sequence of n items from a text. These ‘items’ might be syllables, letters, words, etc. but usually refer to words. Researchers usually vary the size of their n gram depending on their application. Example: “Hi my name is Rick” -> [(Hi, my), (my, name), (name, is), (is, Rick)] would be an example of splitting up a piece of text into 2-grams (called bigrams). We will see how to do this automatically using nltk later in the slides.
• #9: As you can see, the nltk’s built in word tokenizer is able to separate out the sentence into tokens for us to analyze.
• #10: You might say that capitalization or punctuation are good places to start but as you can see proper nouns (Smith) and titles (Mr.) are two examples of common words that break those rules for tokenization This is where built-in tokenizers can help
• #11: As you can see, developers have been hard at work to make sure that tokenizers are able to accurately pick out sentences from a piece of text.
• #12: Why is this useful? Let’s say you didn’t just want to find the most common words, but the most common two-word phrases (like black-eyed peas). N-grams can help with that.
• #13: Good website for interpreting regular expressions: https://regex101.com \s+ / \s+ matches any whitespace character (equal to [\r\n\t\f\v ]) + Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy) [A-Z]['\w]+ Match a single character present in the list below [A-Z] A-Z a single character in the range between A (index 65) and Z (index 90) (case sensitive) Match a single character present in the list below ['\w]+ + Quantifier — Matches between one and unlimited times, as many times as possible, giving back as needed (greedy) ' matches the character ' literally (case sensitive) \w matches any word character (equal to [a-zA-Z0-9_])
• #14: As you can see, the possibilities are endless for tokenization, and the regex you learned in last week’s notes will be very useful if you want full control in how you create your tokens.
• #15: Another example using capitalized words to split the text. This example is a little different though, only the capitalized words were returned. Can you guess why that is? If you guessed that is had something to do with the ‘gaps’ parameter you’d be correct; specifying ‘gaps=True’ tells python that you want to split the text along these patterns to make tokens, otherwise it uses the pattern to find the tokens themselves.
• #16: none
• #17: none
• #18: Stanford’s CoreNLP has different language options
• #19: Why do we want to remove these things? They impede our analysis of the text. To analyze texts, we want a uniform format that we can read in a consistent way. Therefore, we try to remove some of the ‘artifacts’ that are language-specific and don’t contribute to the meaning of the text. The major ways to normalize texts are in the removal of punctuation, capital letters, and numbers.
• #20: Why are we replacing the string with a whitespace instead of nothing? Which is better? Note that the ‘black eyed’ peas portion is different - nothing is good here since you want black and eyed to be grouped together Note that the word ‘I’m’ is different - the whitespace is good here so later ‘I’ can be grouped with other words like ‘I’, ‘me’, etc. when we move into stemming Later in the presentation, we talk about Compound Word Extraction, and that’s a way you can keep ‘black-eyed peas’ together
• #21: Thankfully, strings in python are really easy to send to lowercase, simply use the .lower() function.
• #22: Here are regex knowledge comes in handy because we are able to quickly scan for any digits and words containing digits using the expression ‘\w*\d\w*’. Here is the breakdown of what those characters mean: \d = digits \w = any word character * = 0+ of that character So this removes digits and any words containing digits Notice that the word ‘2lb’ has been removed.
• #23: Lambdas are what as known as “anonymous functions”. They’re meant to be short functions and most of the time they’re single use. The format of a lambda is described above. It is “ lambda input : output “
• #24: Lambdas are cool, but their real power is when you combine them with maps. A map is a python function that takes a function and an iterable as input and iterates over the iterable and applies the function to every object within the iterable. For the example above, you can see that the we applied the ‘square_me’ lambda to every number in the list ‘my_numbers’ The output from a map is a map object, so make sure to cast it to list() type if you want to use the data as a list after.
• #25: Now, let’s look at an example of these tools applied to NLP. As you can see, lambdas and maps make it very easy to remove the numbers in a bunch of texts just by creating one lambda.
• #26: Stop words don’t contribute to the meaning of the text and so for our purposes will just confuse the algorithm. For example, the word ‘some’ doesn’t tell us anything meaningful about what’s going on in this text. On the next slide, there are some of the commonly removed stop words.
• #27: The nltk corpus already comes with a list of stopwords that are commonly removed for text analysis but you can also code your own stopwords manually!
• #28: This is just an early example of CounterVectorizer. We will discuss how a CounterVectorizer is used further in week three. In simple terms, CounterVectorizer convert a collection of text documents to a matrix of token counts. In the example in the slide, the English stops words will be removed from the resulting tokens.
• #29: Lemmatization is closely related to stemming. The difference is that a stemmer operates on a single word without knowledge of the context, and therefore cannot discriminate between words which have different meanings depending on part of speech. However, because stemming does not look at this additional context, stemmers run faster and typically are easier to implement. Depending on your application, the reduced accuracy may not matter.
• #30: As you can see all these different variations of the same word of ‘driving’/’drive’ are reduced to their base form ‘driv’, which is shared by all variations of the word. An example where lemmatization may provide higher accuracy is: The stemmed form of leafs is: leaf The stemmed form of leaves is: leav The lemmatized form of leafs is: leaf The lemmatized form of leaves is: leaf Source: https://towardsdatascience.com/pre-processing-in-natural-language-machine-learning-898a84b8bd47
• #31: Some words change meaning depending on their context in a sentence. For example, the word ‘run’ might be a verb (I like to run in the mornings) or a noun (How did your run go this morning?) depending on how it is used. Thus part of speech tagging may help gain greater insight into the meaning of a text.
• #32: Thankfully, nltk has a built in tagger, so we all we need to do is call this pos_tag from the nltk library. Some POS tags: NN - is a noun NNP – is a proper noun JJ – is an adjective IN – is a preposition VBZ – is a verb, 3rd person sing. present takes
• #33: We find the list with all possible POS tags used by the Natural Language Toolkit (nltk) with nltk.help.upenn_tagset() or nltk.help.upenn_tagset('RB’) for information on a specific tag.
• #34: Named entity recognition is a case where our preprocessing steps would actually hurt us. If we simply removed punctuations and made everything lowercase, we could accidently convert ‘U.S.’ to ‘us’, which could really change the meaning of a document. Being able to extract these ‘entities’ (proper nouns) is a valuable tool, that has a lot of applications. For example, if a news site wants to show all the news that pertains to Chicago or U.S.A, it needs to be able to preserve these words.
• #35: Again, nltk makes our lives easy by providing a built in ‘ne_chunk’ function that is able to detect what the proper nouns in the sentence are (NNP). Notice that it is even able to tell the type of proper noun, and labels ‘James Smith’ as ‘PERSON’ and ‘United States’ as ‘GPE’, which stands for geopolitical entity.
• #36: Compound term extraction again allows us to better preserve the meaning of our text. In the example above, having a text with the word ‘black’, ‘eyed’ and ‘peas’ is different than a text that is about ‘black eyed peas’, which is one distinct concept.
• #37: This is a way to manually do it in NLTK Note that this MWE tokenizer is case sensitive. To make it work better, you’d need to make everything lowercase to begin with.
• #38: none
• #39: Brainstorm as a group. There were a lot of terms that were covered.
• #40: Brainstorm as a group. There were a lot of terms that were covered.