Due: Tuesday, April 23 at 4:30pm.

You must work alone on this assignment.

You will be working with visitor logs from the White House. The attributes in this dataset are described here. Also, you can see a spreadsheet of the data here. Note there are datasets from 2009/2010, 2011, 2012, 2013, 2014, 2015, and 2016. We'll work with the data from 2009/2010.

Your task is to write MapReduce code to generate the following information:

1. A list of the guests who visited at least ten times (task1.py).
2. A list of the ten most frequently-visited staff members (task2.py). The dataset has some obvious anomalies that you will want to address when you answer this question.
3. A list of the guests who visited at least once in both 2009 and 2010 (task3.py).
4. A list of the people who were both guests and staff (in other words, someone whom is visited) in 2009 and / or 2010 (as long as someone visited at any point across the two years, and was visited at any point across the two years, they count, whether or not these two events occurred in the same year) (task4.py).

The terms "guest" and "staff" simply refer to the roles: someone who is a guest is visiting the White House, while a staff member is someone who works there and is being visited. Individuals such as the First Lady, whom you may not consider technically to be "staff," still qualify in this simplistic definition. In other words, these terms are being used to provide intuition, not to create a situation where you are having to make difficult determinations about who constitutes "staff members."

In this writeup, we use the terms "guest" and "staff," but you will also encounter the terms "visitor" and "visitee," which are equivalent, respectively.

For full credit, you must write combiners for each of these tasks whenever it is possible to have a meaningful combiner.

The 2009/2010 dataset is small enough that you will be able to write python code to check your answers. When you can validate the results of a MapReduce calculation using a separate, non-MapReduce approach, this gives you a high degree of confidence that your MapReduce implementation is correct. Of course, it is not particularly compelling to use MapReduce for data sets that do not need it, other than for instructional purposes. But, even for large data sets, performing this kind of double-check on a manageably small subset of the data is wise.

Write your MapReduce versions in python using mrjob. You can do your development on your VirtualBox from CS 122.

To install mrjob on your VirtualBox VM:

sudo pip3 install mrjob

Or, in CSIL:

pip3 install --user mrjob

Unfortunately, due to a regression (a feature that was removed or a bug that was introduced) in recent versions of mrjob, we need to fix a problem with it ourselves. On your VM, type:

sudo subl /usr/local/lib/python3.5/dist-packages/mrjob/sim.py

Or, in CSIL:

subl ~/.local/lib/python3.5/site-packages/mrjob/sim.py

Go to line 379, which should be the return statement for the function _num_reducers. If it is at a slightly different line number, look for this return statement. Then, delete the expression being returned and hard-code in the return value 1/2. In other words, the function should simply say return 1/2. Save your changes and quit.

To run a particular task, if the downloaded White House data is in data.csv:

python3 taskx.py data.csv

for some value of x. You will receive a warning message about "No configs", which you can ignore.

Place your files in a directory called pa1 in your git repository and perform the git push and chisubmit steps you did in 122 (the assignment name in chisubmit is pa1). You should place your code in four python files with names task1.py, task2.py, etc. as above.