← Schedule

HW 4: Dictionaries

CMSC 141: Introduction to Python

Due Sunday, July 19, 11:59pm

A dictionary stores pairs: a key and the value it maps to. Where a list is good at "the thing in position 3," a dictionary is good at "the count for the word the" or "the score for student Maria." This homework uses dictionaries to do a little text analysis: counting words, finding the most common one, and grouping. The last two exercises do that analysis on a real text file instead of a string you type, and write the result back out to a file.

Run git pull, then cd hw4. Your work goes in hw4.py, and the last four exercises read from the provided Shakespeare_Sonnets.txt, which holds the first 100 of Shakespeare's sonnets.

Goals

What this homework builds

This homework builds S10 (dictionaries). S10 is first checked on Quiz 4 and again on the midterm, so use this homework to lock it in. The last two exercises also give you more practice with S11 (files), which is first assessed on Quiz 5.

The counting pattern

The single most useful dictionary move is counting. You look at each item; if you've seen it before, add one to its count; if not, start it at one.

counts = {}
for word in ["a", "b", "a"]:
    if word in counts:
        counts[word] = counts[word] + 1
    else:
        counts[word] = 1
# counts is now {"a": 2, "b": 1}

counts[word] reads or sets the value for a key. if word in counts checks whether a key is already there. You'll reuse this shape in almost every exercise.

Exercise 1: normalize

Before counting words, text needs cleaning so that "Hello," and "hello" count as the same word. Write normalize(text) that returns a new string in lowercase with every character that isn't a letter or a space removed. Loop over text.lower() and keep a character only if ch.isalpha() or ch == " ".

>>> normalize("Hello, World!")
'hello world'

Exercise 2: word_counts

Write word_counts(text) that returns a dictionary mapping each word in text to the number of times it appears. Split the text into words with text.split(), which breaks on whitespace, then apply the counting pattern.

>>> word_counts("a b a c a b")
{'a': 3, 'b': 2, 'c': 1}

Exercise 3: most_common

Write most_common(counts) that takes a dictionary of word counts (like the one from Exercise 2) and returns the word with the highest count. Loop over the keys, keeping track of the best word and its count as you go. (If there's a tie, returning any of the tied words is fine.)

>>> most_common({"a": 3, "b": 2, "c": 1})
'a'

Exercise 4: count_first_letters

Write count_first_letters(words) that takes a list of words and returns a dictionary mapping each first letter to the number of words that start with it. This is grouping: the key is the first character, word[0], and the value is a running count.

>>> count_first_letters(["apple", "ant", "bee"])
{'a': 2, 'b': 1}

Exercise 5: merge_counts

Write merge_counts(d1, d2) that takes two count dictionaries and returns a new dictionary with their counts combined. A word in both should have its counts added; a word in only one keeps its count. Don't change d1 or d2; build and return a new dictionary.

>>> merge_counts({"a": 1, "b": 2}, {"b": 3, "c": 1})
{'a': 1, 'b': 5, 'c': 1}

From a file to a dictionary

Everything above worked on a string you passed in. Now the text comes from a file: Shakespeare_Sonnets.txt, which holds the first 100 of Shakespeare's sonnets. Open it with with, the way we did in class, and loop over it one line at a time:

with open("Shakespeare_Sonnets.txt") as f:
    for line in f:
        ...

Look at how the file is laid out. Each sonnet begins with a line that is just its number, then the sonnet's lines, then a blank line before the next number:

1

From fairest creatures we desire increase,
That thereby beauty's rose might never die,
...

You are going to turn that into a dictionary: the key is the sonnet's number, and the value is a list of that sonnet's lines. Then sonnets[1] hands you back the lines of Sonnet 1 as a list of strings, and sonnets[18] gives you Sonnet 18.

One thing to know about normalize: it keeps only letters and spaces, so it drops the apostrophes and hyphens inside words. beauty's becomes beautys and self-substantial becomes selfsubstantial. That is fine for counting, as long as every word is treated the same way.

Exercise 6: read_sonnets

Write read_sonnets(filename) that reads the file and returns a dictionary mapping each sonnet number to a list of its lines. Open the file with with open(filename) as f: and loop over it with for line in f:. For each line, first .strip() it to remove the trailing newline and any surrounding spaces, and skip it if it is now empty. If the stripped line is all digits, test it with .isdigit(), then it is a sonnet's number: turn it into an integer with int(...) and start a new empty list for it in the dictionary. Otherwise the line belongs to the sonnet you are currently reading, so append it to that sonnet's list.

>>> sonnets = read_sonnets("Shakespeare_Sonnets.txt")
>>> len(sonnets)
100
>>> sonnets[1][0]
'From fairest creatures we desire increase,'

(Most sonnets have 14 lines, but not all of them do, so don't rely on a fixed count.)

Exercise 7: sonnet_word_count

Now that the sonnets are in a dictionary, you can pull any one of them out by its number. Write sonnet_word_count(sonnets, n) that returns the total number of words in sonnet n. Look up sonnets[n] to get that sonnet's list of lines, join them into one string with " ".join(...), then reuse normalize and word_counts and add up the values of the result.

>>> sonnet_word_count(sonnets, 1)
105

Exercise 8: corpus_word_counts

Write corpus_word_counts(sonnets) that returns a single word-count dictionary covering every word across all 100 sonnets. Loop over the sonnets' values (each value is a list of lines), gather all of those lines into one string, and reuse normalize and word_counts rather than rewriting them. Because the result is an ordinary count dictionary, most_common works on it directly.

>>> counts = corpus_word_counts(sonnets)
>>> counts["and"]
321
>>> most_common(counts)
'and'

Exercise 9: write_word_report

Now write a result back out. write_word_report(in_filename, out_filename) reads the sonnets from in_filename, builds the word counts across all of them, and writes a two-line report to out_filename: the total number of words, then the most common word. Reuse read_sonnets and corpus_word_counts to do the reading and counting. Open a file for writing by passing "w" as a second argument to open, and end each written line with "\n":

with open(out_filename, "w") as out:
    out.write("total words: " + str(total) + "\n")

Get the total by adding up the values in the count dictionary, and the most common word from most_common. After running write_word_report("Shakespeare_Sonnets.txt", "report.txt"), the file report.txt should contain:

total words: 11362
most common: and

Requirements summary

In hw4.py, all nine functions return the correct values: normalize, word_counts, most_common, count_first_letters, merge_counts, read_sonnets, sonnet_word_count, corpus_word_counts, and write_word_report. merge_counts must not modify its inputs, read_sonnets must key each sonnet by its number, and write_word_report must write the two report lines in order. Run pytest from the hw4 directory.

Questionnaire

Fill out QUESTIONS-04.txt and submit it with your code. You won't be able to get an S on the homework without it.

Submission

Submit hw4.py on Gradescope under "Homework 4." Pushing your GitHub repo with a finished QUESTIONS-04.txt is sufficient to submit the questionnaire.