INFX 141 / CS 121 • DAVID G. KAY • UC IRVINE • WINTER 2015

Assignment #4
Text Analysis and Indexing

In this assignment you will be exploring a corpus of Email messages about the Enron scandal. The corpus consists of the messages that were sent by or to Enron executives in the months preceding the scandal. It's a 400+MB file, tarred and gzipped, that you can download from A description of the data is available at You are going to play the role of a data analyst hired to examine the evidence by doing some simple fact-finding and data processing.

General specifications:

Part 1. Context (20 points)

Find out about the Enron scandal, for example by finding documentaries on YouTube or text articles online. (One good one, called "The Smartest Guys in the Room," is reportedly available on Netflix.)

Create PDF document named context.pdf that includes:

  1. A concise summary of the scandal (maximum half a page)
  2. A list of each main player with their organizational affiliation, title (if any), and role in the scandal
  3. The year in which the main events unfolded

Part 2. Quantify the data (40 points)

Unzip and untar the data file; if you don't know how to do this, find out. Then quantify the evidence by finding the answers to these questions (keeping notes on the steps you take):

  1. Create a PDF file called quant.pdf that includes answers to those questions and, for each, a description of the process you followed to find those answers. If you used scripts or programs for this part, include them in a zipped folder called part2.
  2. How many people are targeted in this data set? (We're just asking about the folder structure, not about the people mentioned in the emails themselves.)
  3. How many individual data files are we dealing with?
  4. How many messages were sent by these people in total? (Explain how you're interpreting the term "sent.")
  5. How many messages were sitting in these people's Inboxes in total? (Explain how you're determining what counts as an "inbox.")
  6. Who are the 10 people with the largest number of data files?

Part 3 [Developer]. Index the data (40 points)

Create inverted indices for the entire set of data files in the manner explained below. Here are some general notes about these indices:

  1. (Required) Create an index that uses human-readable terms and document identifiers. For example:
    amendment   allen-p/_sent_mail/465.:1:34    stclair-c/sent/993.:5:45,60,76,84,100
    Call this index file index_plain.txt.

  2. (Extra credit, maximum 10 points) Create a second index that uses some encoding of, at least, terms and document identifiers in a way that decreases the size of the index file significantly (at least 20%). You may also compress the position information. Call this index file index_compressed.txt. [You can create this second index by parsing the files again or create it from the first index.]

Part 3 [Analyst]. Estimation (40 points)

Create a PDF document called estimation.pdf that includes responses to the following:

  1. Find all documents that have the terms "conflict of interest", independent of capitalization. Explain what you did to get your answer.
  2. Estimate how many distinct terms are in this data set. Explain the reasoning behind your estimate. If you used scripts or programs, explain what they do.
  3. With respect to the index_plain.txt file described in Part 3 [Developer] above, estimate the size of that file in megabytes. State all of your assumptions and your calculations. You may use scripts, programs, or other tools to help create your estimate, but of course you must disclose what you used.

Submitting your assignment: Your will submit your work via Checkmate. For groups of two or three, just one of you should submit all parts of the assignment; the names of all group members must appear near the top of every submitted file.
First, submit your context.pdf file from Part 1.
Second, submit your quant.pdf file and your zipped part2 folder from Part 2.
Third, for Part 3 [Developer], submit your index_plain.txt file, optionally your index_compressed.txt file, a zipped folder called programs containing your program(s), and optionally (for those who do part 2) a fille called encoding.pdf that explains your encoding.
Third, for Part 3 [Analyst], submit your estimation.pdf file and optionally a zipped folder called programs containing any [non-mainstream, non-public] code you wrote or used for this part.

David G. Kay,
Sunday, February 15, 2015 2:55 PM