Summary of the content in the Data Scientist Essentials course, Part 6: Sentiment analysis on text data

1. Methods for managing text data

Data comes in several types, including:
– Structured data: Data stored in a database.
– Semi-Structured data: XML, JSON, CSV, and some NoSQL databases.
– Unstructured data: Text, photographs, video, and audio.

Managing text data is complex in terms of data preprocessing due to its variety of aspects, including spelling errors, misinterpretation (positive or negative), large amounts of text in a document, and sometimes multiple files considered as a single document 

Preparing data for model building and converting text data into numerical formats in a table can be done in several ways, as follows:

Bag of Words is a method for isolating and counting words that appear in a sentence

• Binary representation: Enter 1 if the word exists; enter 0 if the word is not present
• Term Frequency (TF) counts the number of times a word occurs in a document and displays that number. Sometimes, it's impossible to find the exact word that occurs in the document, resulting in a large number of entries showing 0
• Inverse Document Frequency (IDF) counts the number of times a word occurs in all the documents of interest
• Term Frequency-Inverse Document Frequency (TF-IDF) uses two methods: TF and IDF

If the bag of words method is used, interpretation problems may arise. Therefore, N-gram sequences can help solve this problem

• N-gram sequences count the number of words and add the desired number of words, available in single-word, two-word, and three-word sequences

Adding a feature set requires developing a feature selection method to reduce the number of features, selecting only the most important ones. This will also require more memory and processing space

Speaker: Dr. Phaisan Phadungwiang,
R&D BI Co., Ltd.
Compiled by: Nuchanat Rongrong
www.rdbi.co.th
#Consultants
#Data Scientist
#Big Data Analytics
#text analysis
#Methods for managing text data

(Includes articles on Data Science Essentials)