scala - Iterator behaviour in flink reduceGroup -


i creating system should handle huge amount of data , need understand how reduce group operator works

i have dataset apply groupby , subsequently reducegroup how iterator passed reducegroup function behave? lazy iterator loads data when requested or eager 1 prepares data in memory when created?

i using scala api in flink 0.9 milestone1

flink performs group-by groupreduce using sort operator. sort operator receives memory budget sorting. long data fits budget, sort happen in-memory. otherwise, sort becomes external merge-sort , spills disk. flink reads sorted data stream , applies groupreduce function "on-the-fly". data of group not read in-memory before function applied. hence, can process large groups if user-function not materialize group records itself.


Comments

Popular posts from this blog

android - How to save instance state of selected radiobutton on menu -

python 3 IndexError: list index out of range -

IF statement in MySQL trigger -