Mining Text Data: An Introduction Data Mining /

Mining Text Data: An Introduction Data Mining /

Mining Text Data: An Introduction Data Mining / Knowledge Discovery Structured Data omeLoan ( oanee: Frank Rizzo ender: MWF gency: Lake View mount: $200,000 erm: 15 years 01/27/20 Multimedia Free Text Hypertext Frank Rizzo bought his home from Lake View Real Estate in 1992. He paid $200,000 under a15-year loan Loans($200K,[map],...) from MW Financial. Frank Rizzo Bought this home from Lake View Real Estate In 1992.

... Data Mining: Principles and Algorit 1 Bag-of-Tokens Approaches Documents Token Sets Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal. Now we are engaged in a great civil war, testing whether that nation, or Feature Extraction nation 5 civil - 1 war 2 men 2 died 4 people 5 Liberty 1 God 1

Loses all order-specific information! Severely limits context! 01/27/20 Data Mining: Principles and Algorit 2 Natural Language Processing A dog is chasing a boy on the playground Det Noun Aux Noun Phrase Verb Complex Verb Semantic analysis Dog(d1). Boy(b1). Playground(p1). Chasing(d1,b1,p1). + Det Noun Prep Det Noun Noun Phrase Noun Phrase

Lexical analysis (part-of-speech tagging) Prep Phrase Verb Phrase Syntactic analysis (Parsing) Verb Phrase Sentence Scared(x) if Chasing(_,x,_). Scared(b1) Inference (Taken from ChengXiang Zhai, CS 397cxz Fall 2003) 01/27/20 Data Mining: Principles and Algorit A person saying this may be reminding another person to get the dog back Pragmatic analysis (speech act) 3

General NLPToo Difficult! Word-level ambiguity design can be a noun or a verb (Ambiguous POS) root has multiple meanings (Ambiguous sense) Syntactic ambiguity natural language processing (Modification) A man saw a boy with a telescope. (PP Attachment) Anaphora resolution John persuaded Bill to buy a TV for himself. (himself = John or Bill?) Presupposition He has quit smoking. implies that he smoked before. Humans rely on context to interpret (when possible). This context may extend beyond a given document! (Taken from ChengXiang Zhai, CS 397cxz Fall 2003) 01/27/20 Data Mining: Principles and Algorit 4

Shallow Linguistics Progress on Useful Sub-Goals: English Lexicon Part-of-Speech Tagging Word Sense Disambiguation Phrase Detection / Parsing 01/27/20 Data Mining: Principles and Algorit 5 WordNet An extensive lexical network for the English language Contains over 138,838 words. Several graphs, one for each part-of-speech. Synsets (synonym sets), each defining a semantic sense. Relationship information (antonym, hyponym, meronym ) Downloadable for free (UNIX, Windows) Expanding to other languages (Global WordNet Association) Funded >$3 million, mainly government (translation interest) Founder George Miller, National Medal of Science, 1991. moist watery parched wet dry damp

anhydrous arid synonym 01/27/20 Data Mining: Principles and Algorit antonym 6 Part-of-Speech Tagging This Det sentence N Training data (Annotated text) serves as an example of V1 This is a new sentence. P Det N POS Tagger

P annotated V2 text N This is a new Det Aux Det Adj sentence. N Pick the most sequence. p ( w1 likely ,..., wk , ttag 1 ,..., t k ) p(t1 | w1 )... p(tk | wk ) p( w1 )... p( wk ) p ( w1 ,..., wk , t1 ,..., tk ) k Independent assignment p( wi | ti ) p (ti | ti 1 ) Most common tag p (t1 | w1 )... p (tk | wk ) p(iw11 )... p ( wk ) k p ( wi | ti ) p(ti | ti 1 ) Partial dependency i 1 (HMM)

(Adapted from ChengXiang Zhai, CS 397cxz Fall 2003) 01/27/20 DataMining: Principles and Algorit 7 Word Sense Disambiguation ? The difficulties of computational linguistics are rooted in ambiguity. N Aux V P N Supervised Learning Features: Neighboring POS tags (N Aux V P N) Neighboring words (linguistics are rooted in ambiguity) Stemmed form (root) Dictionary/Thesaurus entries of neighboring words High co-occurrence words (plant, tree, origin,) Other senses of word within discourse Algorithms: Rule-based Learning (e.g. IG guided) Statistical Learning (i.e. Nave Bayes) Unsupervised Learning (i.e. Nearest Neighbor) 01/27/20 Data Mining: Principles and Algorit 8

Parsing Choose most likely parse tree Grammar Lexicon V chasing 0.01 Aux is N dog 0.003 N boy N playground Det the Det a P on Probability of this tree=0.000015 NP Probabilistic CFG 1.0 S NP VP NP Det BNP 0.3 0.4 NP BNP 0.3 NP NP PP BNP N VP V VP Aux V NP

VP VP PP PP P NP 1.0 S VP Det BNP A N Aux dog ... VP PP V NP is chasing P NP

on a boy the playground Probability of this tree=0.000011 S NP Det A VP BNP N Aux is dog (Adapted from ChengXiang Zhai, CS 397cxz Fall 2003) 01/27/20 DataMining: Principles and Algorit NP V chasing NP a boy

PP P NP on the playground 9 Obstacles Ambiguity A man saw a boy with a telescope. Computational Intensity Imposes a context horizon. Text Mining NLP Approach: 1. Locate promising fragments using fast IR methods (bag-of-tokens). 2. Only apply slow NLP techniques to promising fragments. 01/27/20 Data Mining: Principles and Algorit 10 Text Databases and IR Text databases (document databases)

Large collections of documents from various sources: news articles, research papers, books, digital libraries, e-mail messages, and Web pages, library database, etc. Data stored is usually semi-structured Traditional information retrieval techniques become inadequate for the increasingly vast amounts of text data Information retrieval A field developed in parallel with database systems Information is organized into (a large number of) documents Information retrieval problem: locating relevant documents based on user input, such as keywords or example documents 01/27/20 Data Mining: Principles and Algorit 11 Information Retrieval Typical IR systems

Online library catalogs Online document management systems Information retrieval vs. database systems Some DB problems are not present in IR, e.g., update, transaction management, complex objects Some IR problems are not addressed well in DBMS, e.g., unstructured documents, approximate search using keywords and relevance 01/27/20 Data Mining: Principles and Algorit 12 Basic Measures for Text Retrieval Relevant Relevant & Retrieved Retrieved

All Documents Precision: the percentage of retrieved documents that are in fact relevant to the query (i.e., correct responses) | {Relevant} {Retrieved } | precision | {Retrieved} | Recall: the percentage of documents that are relevant to the query and were, in fact, retrieved | {Relevant} {Retrieved } | precision | {Relevant} | 01/27/20 Data Mining: Principles and Algorit 13 Information Retrieval Techniques Basic Concepts A document can be described by a set of representative keywords called index terms. Different index terms have varying relevance when used to describe document contents. This effect is captured through the assignment

of numerical weights to each index term of a document. (e.g.: frequency, tf-idf) DBMS Analogy Index Terms Attributes Weights Attribute Values 01/27/20 Data Mining: Principles and Algorit 14 Information Retrieval Techniques Index Terms (Attribute) Selection: Stop list Word stem Index terms weighting methods Terms Documents Frequency Matrices Information Retrieval Models: Boolean Model Vector Model Probabilistic Model 01/27/20 Data Mining: Principles and Algorit 15 Boolean Model

Consider that index terms are either present or absent in a document As a result, the index term weights are assumed to be all binaries A query is composed of index terms linked by three connectives: not, and, and or 01/27/20 e.g.: car and repair, plane or airplane The Boolean model predicts that each document is either relevant or non-relevant based on the match of a document to the query Data Mining: Principles and Algorit 16 Keyword-Based Retrieval

A document is represented by a string, which can be identified by a set of keywords Queries may use expressions of keywords E.g., car and repair shop, tea or coffee, DBMS but not Oracle Queries and retrieval should consider synonyms, e.g., repair and maintenance Major difficulties of the model Synonymy: A keyword T does not appear anywhere in the document, even though the document is closely related to T, e.g., data mining Polysemy: The same keyword may mean different things in different contexts, e.g., mining 01/27/20 Data Mining: Principles and Algorit 17 Similarity-Based Retrieval in Text Data Finds similar documents based on a set of common keywords Answer should be based on the degree of relevance based on the nearness of the keywords,

relative frequency of the keywords, etc. Basic techniques Stop list Set of words that are deemed irrelevant, even though they may appear frequently E.g., a, the, of, for, to, with, etc. Stop lists may vary when document set varies 01/27/20 Data Mining: Principles and Algorit 18 Similarity-Based Retrieval in Text Data 01/27/20 Word stem Several words are small syntactic variants of each other since they share a common word stem E.g., drug, drugs, drugged A term frequency table

Each entry frequent_table(i, j) = # of occurrences of the word ti in document di Usually, the ratio instead of the absolute number of occurrences is used Similarity metrics: measure the closeness of a document to a query (a set of keywords) Relative term occurrences sim(v , v ) v1 v2 1 2 | v1 || v2 | Cosine distance: Data Mining: Principles and Algorit 19 Vector Space Model Documents and user queries are represented as mdimensional vectors, where m is the total number of index terms in the document collection. The degree of similarity of the document d with regard to the query q is calculated as the correlation between the vectors that represent them, using measures such as the Euclidian distance or the cosine of the angle between these two vectors. 01/27/20 Data Mining: Principles and Algorit 20

Latent Semantic Indexing Basic idea Similar documents have similar word frequencies Difficulty: the size of the term frequency matrix is very large Use a singular value decomposition (SVD) techniques to reduce the size of frequency table Retain the K most significant rows of the frequency table Method Create a term x document weighted frequency matrix A SVD construction: A = U * S * V Define K and obtain Uk ,, Sk , and Vk. Create query vector q . Project q into the term-document space: Dq = q * U k * Sk-1

Calculate similarities: cos = Dq . D / ||Dq|| * ||D|| 01/27/20 Data Mining: Principles and Algorit 21 Probabilistic Model Basic assumption: Given a user query, there is a set of documents which contains exactly the relevant documents and no other (ideal answer set) Querying process as a process of specifying the properties of an ideal answer set. Since these properties are not known at query time, an initial guess is made This initial guess allows the generation of a preliminary probabilistic description of the ideal answer set which is used to retrieve the first set of documents An interaction with the user is then initiated with the purpose of improving the probabilistic description of the answer set 01/27/20 Data Mining: Principles and Algorit

22 Types of Text Data Mining Keyword-based association analysis Automatic document classification Similarity detection Cluster documents by a common author Cluster documents containing information from a common source Link analysis: unusual correlation between entities Sequence analysis: predicting a recurring event Anomaly detection: find information that violates usual patterns Hypertext analysis Patterns in anchors/links Anchor text correlations with linked objects 01/27/20 Data Mining: Principles and Algorit 23

Keyword-Based Association Analysis Motivation Collect sets of keywords or terms that occur frequently together and then find the association or correlation relationships among them Association Analysis Process Preprocess the text data by parsing, stemming, removing stop words, etc. Evoke association mining algorithms View a set of keywords in the document as a set of items in the transaction Term level association mining 01/27/20

Consider each document as a transaction No need for human effort in tagging documents The number of meaningless results and the execution time is greatly reduced Data Mining: Principles and Algorit 24 Text Classification Motivation Automatic classification for the large number of on-line text documents (Web pages, e-mails, corporate intranets, etc.) Classification Process Data preprocessing Definition of training set and test sets Creation of the classification model using the selected classification algorithm Classification model validation Classification of new/unknown text documents Text document classification differs from the classification of relational data Document databases are not structured according to attribute-value pairs

01/27/20 Data Mining: Principles and Algorit 25 Text Classification(2) Classification Algorithms: Support Vector Machines K-Nearest Neighbors Nave Bayes Neural Networks Decision Trees Association rule-based Boosting 01/27/20 Data Mining: Principles and Algorit 26 Document Clustering Motivation Automatically group related documents based on their contents No predetermined training sets or taxonomies Generate a taxonomy at runtime

Clustering Process Data preprocessing: remove stop words, stem, feature extraction, lexical analysis, etc. Hierarchical clustering: compute similarities applying clustering algorithms. Model-Based clustering (Neural Network Approach): clusters are represented by exemplars. (e.g.: SOM) 01/27/20 Data Mining: Principles and Algorit 27 Text Categorization Pre-given categories and labeled document examples (Categories may form hierarchy) Classify new documents A standard classification (supervised learning ) problem Sports Categorization System Business Education Sports

Business Science Education 01/27/20 Data Mining: Principles and Algorit 28 Applications News article classification Automatic email filtering Webpage classification Word sense disambiguation 01/27/20 Data Mining: Principles and Algorit 29 Categorization Methods

Manual: Typically rule-based Does not scale up (labor-intensive, rule inconsistency) May be appropriate for special data on a particular domain Automatic: Typically exploiting machine learning techniques Vector space model based Probabilistic or generative model based 01/27/20 Prototype-based (Rocchio) K-nearest neighbor (KNN) Decision-tree (learn rules) Neural Networks (learn non-linear classifier) Support Vector Machines (SVM) Nave Bayes classifier Data Mining: Principles and Algorit 30 Vector Space Model

Represent a doc by a term vector Term: basic concept, e.g., word or phrase Each term defines one dimension N terms define a N-dimensional space Element of vector corresponds to term weight E.g., d = (x1,,xN), xi is importance of term i New document is assigned to the most likely category based on vector similarity. 01/27/20 Data Mining: Principles and Algorit 31 VS Model: Illustration

Starbucks C2 Category 2 Category 3 C3 new doc Microsoft 01/27/20 Java C1 Category 1 Data Mining: Principles and Algorit 32 How to Assign Weights Two-fold heuristics based on frequency TF (Term frequency) IDF (Inverse document frequency)

01/27/20 More frequent within a document more relevant to semantics e.g., query vs. commercial Less frequent among documents more discriminative e.g. algebra vs. science Data Mining: Principles and Algorit 33 TF Weighting Weighting: More frequent => more relevant to topic e.g. query vs. commercial Raw TF= f(t,d): how many times term t appears in doc d Normalization: Document length varies => relative frequency

preferred 01/27/20 e.g., Maximum frequency normalization Data Mining: Principles and Algorit 34 IDF Weighting Ideas: Less frequent among documents more discriminative Formula: n total number of docs # docs with term t appearing k (the DF document frequency) 01/27/20 Data Mining: Principles and Algorit 35 TF-IDF Weighting

TF-IDF weighting : weight(t, d) = TF(t, d) * IDF(t) Freqent within doc high tf high weight Selective among docs high idf high weight Recall VS model Each selected term represents one dimension Each doc is represented by a feature vector Its t-term coordinate of document d is the TF-IDF weight This is more reasonable Just for illustration Many complex and more effective weighting variants exist in practice 01/27/20 Data Mining: Principles and Algorit 36 How to Measure Similarity? Given two document

Similarity definition dot product 01/27/20 normalized dot product (or cosine) Data Mining: Principles and Algorit 37 Illustrative Example text mining search engine text doc1 travel text doc2 Sim(newdoc,doc1)=4.8*2.4+4.5*4.5 Sim(newdoc,doc2)=2.4*2.4 Sim(newdoc,doc3)=0 map travel

text IDF(faked) 2.4 doc3 To whom is newdoc more similar? government president congress 01/27/20 mining travel 4.5 2.8 doc1 doc2 doc3 2(4.8) 1(4.5) 1(2.4 ) newdoc 1(2.4) 1(4.5) map search engine govern president congress 3.3 2.1 5.4 2.2

3.2 4.3 1(2.1) 1(5.4) 2 (5.6) 1(3.3) 1 (2.2) 1(3.2) Data Mining: Principles and Algorit 1(4.3) 38 VS Model-Based Classifiers What do we have so far? A feature space with similarity measure This is a classic supervised learning problem Search for an approximation to classification hyper plane VS model based classifiers K-NN Decision tree based Neural networks Support vector machine

01/27/20 Data Mining: Principles and Algorit 39 Categorization Methods Vector space model K-NN Decision tree Neural network Support vector machine Probabilistic model Nave Bayes classifier

Many, many others and variants exist [F.S. 02] 01/27/20 e.g. Bim, Nb, Ind, Swap-1, LLSF, Widrow-Hoff, Rocchio, Gis-W, Data Mining: Principles and Algorit 40 Evaluations Effectiveness measure Classic: Precision & Recall 01/27/20 Precision Recall Data Mining: Principles and Algorit 41 Evaluation (cont) Benchmarks

Classic: Reuters collection A set of newswire stories classified under categories related to economics. Effectiveness 01/27/20 Difficulties of strict comparison different parameter setting different split (or selection) between training and testing various optimizations However widely recognizable Best: Boosting-based committee classifier & SVM

Worst: Nave Bayes classifier Need to consider other factors, especially efficiency Data Mining: Principles and Algorit 42 Summary: Text Categorization Wide application domain Comparable effectiveness to professionals Manual TC is not 100% and unlikely to improve substantially. A.T.C. is growing at a steady pace Prospects and extensions 01/27/20

Very noisy text, such as text from O.C.R. Speech transcripts Data Mining: Principles and Algorit 43

Recently Viewed Presentations

  • Java Methods A & AB

    Java Methods A & AB

    Collection: (hmm...) any collection of elements Overview (cont'd) Collection, Iterator Lists, ListIterator List ArrayList LinkedList Stack Queue, PriorityQueue Sets Set TreeSet HashSet Maps Map TreeMap HashMap Overview (cont'd) Overview (cont'd) A collection holds references to objects (but we say informally...
  •  Essential Question:  How did the end of World

    Essential Question: How did the end of World

    USWA Agenda for Unit 3: "America in the 1920s" notes Today's HW: 20.1 and 20.2 Unit 3 Test: TBA 1920s: Sports Sports was a popular form of entrainment in the 1920s as Americans gained more leisure time and income Baseball,...
  • Analog and RF Circuit Testing - Auburn University

    Analog and RF Circuit Testing - Auburn University

    That is, we label voltages close to the supply voltage as logic HIGH, and voltage close to zero as logic LOW. If you were to probe a node using an oscilloscope, you'd still see continuous time signals out of the...
  • GCSE Physical Education Prepare for the first assessment in 2018

    GCSE Physical Education Prepare for the first assessment in 2018

    level; NB. This refers to general aspects of the process, not specific elements of content. • Having provided advice at a general level, allow candidates to revise and re-draft work.. General advice of this nature . does not . need...
  • Les theories de l'apprentissage

    Les theories de l'apprentissage

    Le rat trouve tout de suite sa nourriture, même en étant entré par une autre entrée. Tolman montre ici que le rat a décoder l'organisation spatiale du labyrinthe, a développé une carte cognitive (représentation spatiale d'une réalité), au lieu d'un...
  • By Tommy Boss

    By Tommy Boss

    "To say that a work of art is good, but incomprehensible to the majority of men, is the same as saying of some kind of food that it is very good but that most people can't eat it."-Leo Tolstoy. The...
  • Bullying 101: - PACER

    Bullying 101: - PACER

    Conflict vs. Bullying: What's the Difference? PACER's National Bullying Prevention Center ©2018. Conflict is a disagreement or argument in which both sides express their views. Bullying is behavior with intention to hurt and the other person often doesn't feel safe...
  • How English auction maximizes profits in art auctions

    How English auction maximizes profits in art auctions

    Art is therefore unlike other multiple identical goods (e.g. land, car plate…etc.) Art is more individualized, one original copy by one artist. ... Art auctions: A survey of empirical studies, National Bureau of Economics Research Working Paper 8997, 2002.