Writing to BigQuery from Dataflow - JSON files are not deleted when a job finishes -


one of our dataflow jobs writes output bigquery. understanding of how implemented under-the-hood, dataflow writes results (sharded) in json format gcs, , kicks off bigquery load job import data.

however, we've noticed json files not deleted after job regardless of whether succeeds or fails. there no warning or suggestion in error message files not deleted. when noticed this, had @ our bucket , had hundreds of large json files failed jobs (mostly during development).

i have thought dataflow should handle cleanup, if job fails, , when succeeds files should deleted leaving these files around after job has finished incurs significant storage costs!

is bug?

example job id of job "succeeded" left hundreds of large files in gcs: 2015-05-27_18_21_21-8377993823053896089

enter image description here

enter image description here

enter image description here

because still happening decided we'll clean ourselves after pipeline has finished executing. run following command delete not jar or zip:

gsutil ls -p <project_id> gs://<bucket> | grep -v '[zip|jar]$' | xargs -n 1 gsutil -m rm -r

Comments

Post a Comment

Popular posts from this blog

android - MPAndroidChart - How to add Annotations or images to the chart -

i working latest mpandroidchart library , looking way add annotations or images clickable. example: once data reaches value event occurs want able display user - there multiple events can occur annotations (not sure if right word) or image added @ point in chart. any pointers welcome. check documentation of yaxis . focus on setspacetop(...) , setaxismaxvalue(...) methods. for background: bar-shadow , disable calling: barchart.setdrawbarshadow(false) or change color: bardataset.setbarshadowcolor(int color)
Read more

javascript - Add class to another page attribute using URL id - Jquery -

i have attached id url. ex: when click link 1 , has #tab-2 id , takes me landing page. in landing page have find element has #tab-2 id have add class .active . i tried no luck. please 1 correct code. thanks js fiddle link html <div class="menu"> <ul> <li><a href="http://jsfiddle.net/gscyoa0j/1/#tab-2">link 1</a></li> <li><a href="http://jsfiddle.net/gscyoa0j/1/#tab-44">link 2</a></li> <li><a href="http://jsfiddle.net/gscyoa0j/1/#tab-74">link 3</a></li> </ul> </div> landing page <div class="tab"> <ul> <li><a id="tab-2" class="active" >link 1 content</a></li> <li><a id="tab-44" >link 2 content</a></li> <li><a id="tab-74" >link 3 content</a></l
Read more

firefox - Where is 'webgl.osmesalib' parameter? -

i'm trying webgl running on firefox (v37.0.2) via software acceleration mesa. tried following steps have been stated in article: http://www.binarytides.com/enable-webgl-firefox-ubuntu/ however, unable find parameter named ' webgl.osmesalib ' in about:config section? has been removed (or renamed)? if yes, how should instruct firefox use mesa's software acceleration webgl? looks support osmesa removed in favour llvmpipe.
Read more