ud617 - Intro to Hadoop and MapReduce

Programming assignments from Udacity course Intro to Hadoop and MapReduce.

Lesson 4: Combine Datasets

Given a set of forum posts and a set of user info, use map-reduce to merge the datasets based on the shared author_id key.

Given a set of forum posts, use map-reduce to solve the following problems:

For each student, find the hour during which the student has posted the most posts.

For each question, find the length of the post and the average answer length.

Find the Top 10 tags, ordered by the number of questions they appear in.

For each forum thread (i.e. a question node with all its answers and comments), list the students that have posted there.

Udacity provides two datasets compatible with the scripts:

Validation instructions are provided here:

Here is the output generated by the scripts when using the small dataset:

Name		Name	Last commit message	Last commit date
Latest commit History 17 Commits
README.md		README.md
average_length.tsv		average_length.tsv
average_length_mapper.py		average_length_mapper.py
average_length_reducer.py		average_length_reducer.py
combine_datasets_mapper.py		combine_datasets_mapper.py
combine_datasets_reducer.py		combine_datasets_reducer.py
popular_tags.tsv		popular_tags.tsv
popular_tags_mapper.py		popular_tags_mapper.py
popular_tags_reducer.py		popular_tags_reducer.py
student_times.tsv		student_times.tsv
student_times_mapper.py		student_times_mapper.py
student_times_reducer.py		student_times_reducer.py
study_groups.tsv		study_groups.tsv
study_groups_mapper.py		study_groups_mapper.py
study_groups_reducer.py		study_groups_reducer.py