PageRank example getting stuck due to explosion of out/pagerank_data_N

[dimitargnikolov]

I just tried the PageRank example from Chapter 9, but it gets stuck in the reduce phase of a process on the 5th iteration of the algorithm.

Digging a little deeper, it looks like the contents of the pagerank_data_N directories are exploding in size as N increases:

hdfs dfs -du -h out/
5        out/max_diff_1
18       out/max_diff_2
19       out/max_diff_3
18       out/max_diff_4
333.7 K  out/pagerank_data_1
768.6 K  out/pagerank_data_2
9.2 M    out/pagerank_data_3
2.2 G    out/pagerank_data_4
33.8 G   out/pagerank_data_5

Looking further into the contents of pagerank_data_N, it looks like a large number of duplicates end up in the output. I was able to fix by applying DISTINCT to new_pagerank:

new_pagerank = ...
new_pagerank_final = distinct new_pagerank; 
store new_pagerank_final into '$docs_out' using PigStorage('\t');
..