Bilisoly - Practical Text Mining with Perl

WILEY SERIES ON METHODS AND APPLICATIONS IN DATA MINING

Series Editor: Daniel T. Larose

Discovering Knowledge in Data: An Introduction to Data Mining Daniel T. LaRose

Data-Mining on the Web: Uncovering Patterns in Web Content, Structure, and Usage Zdravko Markov and Daniel Larose

Data Mining Methods and Models Daniel Larose

Practical Text Mining with Perl Roger Bilisoly

Copyright 2008 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.
Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic format. For information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data:

Bilisoly, Roger, 1963
Practical text mining with Perl / Roger Bilisoly. p. cm.
Includes bibliographical references and index.
ISBN 978-0-470-17643-6 (cloth)
1. Data mining. 2. Text processing (Computer science) 3. Perl (Computer program language) I. Title.
QA76.9.D343.B45 2008
005.74dc22
2008008144

To my Mom and Dad & all their cats.

List of Figures

Log(Frequency) vs. Log(Rank) for the words in Dickenss A ChristmasCarol.

Plot of the running estimate of the probability of heads for 50 flips.

Plot of the running estimate of the probability of heads for 5000 flips.

Histogram of the proportions of the letter e in 68 Poe short stories based ontable4.l.

Histogram and best fitting normal curve for the proportions of the letter e in 68 Poe short stories.

Plot of the number of types versus the number of tokens for The Unparalleled Adventures of One Hans Pfaall. Data is from program 4.5.Figure adapted from figure 1.1 of Baayen [61 with kind permission from Springer Science and Business Media and the author.

Plot of the mean word frequency against the number of tokens for The Unparalleled Adventures of One Hans Pfaall. Data is from program 4.5. Figure adapted from figure 1.1 of Baayen [61] with kind permission from Springer Science and Business Media and the author.

Plot of the mean word frequency against the number of tokens for The Unparalleled Adventures of One Hans Pfaall and The Black Cat. Figure adapted from figure 1.1 of Baayen [6] with kind permission from Springer Science and Business Media and the author.

The vector (4,3) makes a right triangle if a line segment perpendicular to the x-axis is drawn to the x-axis.

Comparing the frequencies of the word the (on the x-axis) against city (on the y-axis). Note that the y-axis is not to scale: it should be more compressed.

Comparing the logarithms of the frequencies for the words the (on the x-axis) and city (on the y-axis).

Plotting pairs of word counts for the 68 Poe short stories.

Plots of the word counts for the versus of using the 68 Poe short stories.

A two variable data set that has two obvious clusters.

The perpendicular bisector of the line segment from (0,1) to (1,1)divides this plot into two half-planes. The points in each form the two clusters.

. The line splits the data into two groups, and the two centroids are given by the asterisks.

Scatterplot of heRate against sheRate for Poes 68 short stories.

Plot of two short story clusters fitted to the heRate and sheRate data.

Plots of three, four, five, and six short story clusters fitted to the heRate and sheRate data.

Plots of two short story clusters based on eight variables, but only plotted for the two variables heRate and sheRate.

Four more plots showing projections of the two short story clusters found in output 8.7 onto two pronoun rate axes.

Eight principal components split into two short story clusters and projected onto the first two PCs.

A portion of the dendrogram computed in output 8.11, which shows hierarchical clusters for Poes 68 short stories.

The plot of the Voronoi diagram computed in output 8.12.

All four plots have uniform marginal distributions for both the x and y-axes. For problem 8.4.

The dendrogram for the distances between pronouns based on Poes 68 short stories. For problem 8.5.

.

Histogram of the runs of the 10,000 permutations of the names Scrooge and Marley as they appear in A Christmas Carol.

Histogram of the runs of the 10,000 permutations of the names Francois and Perrault as they appear in The Call of the Wild.

List of Tables

Telephone number formats we wish to find with a regex. Here d stands for a digit 0 through 9.

Telephone number input to test regular expression 2.2.

Summary of some of the special characters used by regular expressions with examples of strings that match.

Removing punctuation: a sample of five mistakes made by program 2.4.

Some values of the Perl variable $1 and their effects.

A variety of ways of combining two short sentences.

Sentence segmentation by program 2.8 fails for this sentence.

Defining true and false in Perl.

Comparison of arrays and hashes in Perl.

Proportions of the letter e for 68 Poe short stories, sorted smallest to largest.

.

Counts of four-letter words satisfying each pair of conditions. For problem 4.5.

Preface

What This Book Covers

This book introduces the basic ideas of text mining, which is a group of techniques that extracts useful information from one or more texts. This is a practical book, one that focuses on applications and examples. Although some statistics and mathematics is required, it is kept to a minimum, and what is used is explained.