can we start a new thread in foreachRDD in spark streaming? -


i want start child-thread in foreachrdd.

my situation is:

the job reading hdfs dir continuously, , every 100 batches, want launch model training task (i make snapshot of rdds @ time , start training task. training task takes long time(2 hours), , don't want training task influence reading new batch of data.

is starting new child thread solution? child thread use sparkcontext in main thread , use rdd in main thread?


Comments

Post a Comment

Popular posts from this blog

IF statement in MySQL trigger -

this query code in mysql trigger. if 1=2 select 'yes'; else select 'no'; end if; but has following error: if '1'='2' select 'yes' error code: 1064. have error in sql syntax; check manual corresponds mysql server version right syntax use near 'if '1'='2' select 'yes'' @ line 1 0.000 sec what solution?
Read more

c++ - What does MSC in "// appease MSC" comments mean? -

i run across // appease msc in c++ comments, e.g. here : const int& array::operator[](int offset) const { int mysize = getitssize(); if (offset >= 0 && offset < getitssize()) return ptype[offset]; throw xboundary(); return ptype[offset]; // appease msc! } what msc stand ? the author means microsoft's compiler, more formally known msvc.
Read more

javascript - Blogger related post gadget image Resize s72-c [ Need Expert Help ] -

i run blogger blog , use follow code javascript code (requires no jquery) show related post labels/categories of post. <script type='text/javascript'> var defaultnoimage=&quot;http://1.bp.blogspot.com/-m72rpguntq0/vuokijudn_i/aaaaaaaaboi/lq18sceunsg/w72/favicon-tik.png&quot;; var maxresults=16; var splittercolor=&quot;#d4eaf2&quot;; var relatedpoststitle=&quot;related posts&quot;; </script> <script type='text/javascript'>//<![cdata[ var relatedtitles = new array(); var relatedtitlesnum = 0; var relatedurls = new array(); var thumburl = new array(); function related_results_labels_thumbs(json) { (var = 0; < json.feed.entry.length; i++) { var entry = json.feed.entry[i]; relatedtitles[relatedtitlesnum] = entry.title.$t; try { thumburl[relatedtitlesnum] = entry.media$thumbnail.url } catch (error) {...
Read more