How to deploy and run at a specific time Python scripts on Apache Spark? -


i have set of simple python 2.7 scripts. also, have set of linux nodes. want run @ specific time these scripts on these nodes.

each script may work on each node. script not able run on multiple nodes simultaneously.

so, want complete 3 simple tasks:

  1. to deploy set of scripts.
  2. to run @ specific time main script specific parameters on node.
  3. to result, when script finished.

it seems, able complete first task. have following code snippet:

import urllib import urlparse pyspark import sparkcontext   def path2url(path):     return urlparse.urljoin(         'file:', urllib.pathname2url(path))  master_url = "spark://my-pc:7077" deploy_zip_path = "deploy.zip" sc = sparkcontext(master=("%s" % master_url), appname="job submitter", pyfiles=[path2url("%s" % deploy_zip_path)])

but have problems. code launches tasks. want deploy scripts nodes.

  1. i recommend keeping code deploy pyspark scripts outside of pyspark scripts.

  2. chronos job scheduler runs on apache mesos. spark can run on mesos. chronos runs jobs shell command. so, can run scripts arguments specify. need deploy spark , scripts mesos nodes. then, can run submit spark scripts chronos using spark-submit command.

  3. you store results writing kind of storage mechanism within pyspark scripts. spark has support text files, hdfs, amazon s3, , more. if spark doesn't support storage mechanism need, can use external library does. example, write cassandra in pyspark scripts using cassandra-driver.


Comments

Post a Comment

Popular posts from this blog

IF statement in MySQL trigger -

this query code in mysql trigger. if 1=2 select 'yes'; else select 'no'; end if; but has following error: if '1'='2' select 'yes' error code: 1064. have error in sql syntax; check manual corresponds mysql server version right syntax use near 'if '1'='2' select 'yes'' @ line 1 0.000 sec what solution?
Read more

c++ - What does MSC in "// appease MSC" comments mean? -

i run across // appease msc in c++ comments, e.g. here : const int& array::operator[](int offset) const { int mysize = getitssize(); if (offset >= 0 && offset < getitssize()) return ptype[offset]; throw xboundary(); return ptype[offset]; // appease msc! } what msc stand ? the author means microsoft's compiler, more formally known msvc.
Read more

javascript - Blogger related post gadget image Resize s72-c [ Need Expert Help ] -

i run blogger blog , use follow code javascript code (requires no jquery) show related post labels/categories of post. <script type='text/javascript'> var defaultnoimage=&quot;http://1.bp.blogspot.com/-m72rpguntq0/vuokijudn_i/aaaaaaaaboi/lq18sceunsg/w72/favicon-tik.png&quot;; var maxresults=16; var splittercolor=&quot;#d4eaf2&quot;; var relatedpoststitle=&quot;related posts&quot;; </script> <script type='text/javascript'>//<![cdata[ var relatedtitles = new array(); var relatedtitlesnum = 0; var relatedurls = new array(); var thumburl = new array(); function related_results_labels_thumbs(json) { (var = 0; < json.feed.entry.length; i++) { var entry = json.feed.entry[i]; relatedtitles[relatedtitlesnum] = entry.title.$t; try { thumburl[relatedtitlesnum] = entry.media$thumbnail.url } catch (error) {...
Read more