amazon web services - What controls the number of partitions when reading Parquet files? -


my setup:

two spark clusters. 1 on ec2 , 1 on amazon emr. both spark 1.3.1.

the emr cluster installed emr-bootstrap-actions. ec2 cluster installed spark's default ec2 scripts.

the code:

read folder containing 12 parquet files , count number of partitions

val logs = sqlcontext.parquetfile(“s3n://mylogs/”) logs.rdd.partitions.length

observations:

  • on ec2 code gives me 12 partitions (one per file, makes sense).
  • on emr code gives me 138 (!) partitions.

question:

what controls number of partitions when reading parquet files?

i read exact same folder on s3, exact same spark release. leads me believe there might configuration settings control how partitioning happens. have more info on this?

insights appreciated.

thanks.

update:

it seems many partitions created emr's s3 file system implementation (com.amazon.ws.emr.hadoop.fs.emrfilesystem).

when removing 

<property><name>fs.s3n.impl</name><value>com.amazon.ws.emr.hadoop.fs.emrfilesystem</value></property>

from core-site.xml (hereby reverting hadoop's s3 filesystem), end 12 partitions.

when running emrfilesystem, seems number of partitions can controlled with: 

<property><name>fs.s3n.block.size</name><value>xxx</value></property>

could there cleaner way of controlling # of partitions when using emrfilesystem?


Comments

Post a Comment

Popular posts from this blog

IF statement in MySQL trigger -

this query code in mysql trigger. if 1=2 select 'yes'; else select 'no'; end if; but has following error: if '1'='2' select 'yes' error code: 1064. have error in sql syntax; check manual corresponds mysql server version right syntax use near 'if '1'='2' select 'yes'' @ line 1 0.000 sec what solution?
Read more

c++ - What does MSC in "// appease MSC" comments mean? -

i run across // appease msc in c++ comments, e.g. here : const int& array::operator[](int offset) const { int mysize = getitssize(); if (offset >= 0 && offset < getitssize()) return ptype[offset]; throw xboundary(); return ptype[offset]; // appease msc! } what msc stand ? the author means microsoft's compiler, more formally known msvc.
Read more

javascript - Blogger related post gadget image Resize s72-c [ Need Expert Help ] -

i run blogger blog , use follow code javascript code (requires no jquery) show related post labels/categories of post. <script type='text/javascript'> var defaultnoimage=&quot;http://1.bp.blogspot.com/-m72rpguntq0/vuokijudn_i/aaaaaaaaboi/lq18sceunsg/w72/favicon-tik.png&quot;; var maxresults=16; var splittercolor=&quot;#d4eaf2&quot;; var relatedpoststitle=&quot;related posts&quot;; </script> <script type='text/javascript'>//<![cdata[ var relatedtitles = new array(); var relatedtitlesnum = 0; var relatedurls = new array(); var thumburl = new array(); function related_results_labels_thumbs(json) { (var = 0; < json.feed.entry.length; i++) { var entry = json.feed.entry[i]; relatedtitles[relatedtitlesnum] = entry.title.$t; try { thumburl[relatedtitlesnum] = entry.media$thumbnail.url } catch (error) {...
Read more