scala - Iterator behaviour in flink reduceGroup -


i creating system should handle huge amount of data , need understand how reduce group operator works

i have dataset apply groupby , subsequently reducegroup how iterator passed reducegroup function behave? lazy iterator loads data when requested or eager 1 prepares data in memory when created?

i using scala api in flink 0.9 milestone1

flink performs group-by groupreduce using sort operator. sort operator receives memory budget sorting. long data fits budget, sort happen in-memory. otherwise, sort becomes external merge-sort , spills disk. flink reads sorted data stream , applies groupreduce function "on-the-fly". data of group not read in-memory before function applied. hence, can process large groups if user-function not materialize group records itself.


Comments

Popular posts from this blog

IF statement in MySQL trigger -

c++ - What does MSC in "// appease MSC" comments mean? -

javascript - Blogger related post gadget image Resize s72-c [ Need Expert Help ] -