Program freezes at Map 99% Reduce 33%
I am running a hadoop program processing Tera Byte size data. The code was test successfully on a small sample (100G) and it worked. However, when trying it on the full problem, the program freezes...
View ArticleProgram freezes at Map 99% Reduce 33%
Shi, The key here is the 99% done mapper. Nothing can move on until all mappers complete. Is it possible your data in the larger set has an incomplete record or some such at the end? Kevin...
View ArticleProgram freezes at Map 99% Reduce 33%
Hi Kevin, thanks for reply. I could hardly imagine an example of incomplete record. The mapper is very simple, just reading line by line as Strings, splitting the line by tab, and outputting a Text...
View ArticleProgram freezes at Map 99% Reduce 33%
Shi, This states "Of course, the framework discards the sub-directory of unsuccessful task-attempts." http://hadoop-karma.blogspot.com/2011/01/hadoop-cookbook-how-to-write.ht ml So yes, the missing...
View ArticleProgram freezes at Map 99% Reduce 33%
Hi Kevin, thanks for the suggestion. I think I found the problem, because my code is a chained map / reduce. In the previous iteration there is a .lzo_deflate output which is 40 times larger than other...
View ArticleProgram freezes at Map 99% Reduce 33%
Good. Data skew should not look stuck. Try sending status updates so at least you can tell one mapper is still busy. Yes, adding data or including another field into the key can help reduce data skew....
View Article