Dataflow zombie jobs - stuck in "Not Started" state -
all of our dataflow jobs have stopped working. show "not started".
when kicked off 1 job appears have spawned numerous other jobs, hanging.
is service broken?
the list of job id's:
- 2015-05-12_04_15_09-9449594780471772631
- 2015-05-12_04_11_43-2832089474782567234
- 2015-05-12_04_11_10-7703117482304158028
- 2015-05-12_04_06_52-8133922783285731870
- 2015-05-12_04_06_09-14187812688860505584
- 2015-05-12_04_05_32-10296794562342944020
- 2015-05-12_04_04_58-17815218306022481742
- 2015-05-12_04_04_26-1948202417139012084
- 2015-05-12_04_03_55-5718237782405777885
- 2015-05-12_04_03_23-8040675812721773662
44227 [main] info com.google.cloud.dataflow.sdk.util.packageutil - uploading pipelineoptions.filestostage complete: 1 files newly uploaded, 77 files cached dataflow sdk version: 0.4.150414 446168 [main] warn com.google.cloud.dataflow.sdk.util.retryhttprequestinitializer - request failed code 429, not retry: https://dataflow.googleapis.com/v1b3/projects/gdfp-xxx/jobs disconnected target vm, address: '127.0.0.1:54217', transport: 'socket' 446171 [main] error com.tls.cdf.dfp.dfpdenormalizationclouddataflowjob - exception encountered while trying execute "dfp denormalization job" java.lang.runtimeexception: failed create workflow job: (40153232ba863405): workflow not created. please try again in few minutes. if still unable create job please contact customer support. causes: (40153232ba8632a6): job not created. please try again in few minutes. if service still isn't working please contact customer support. causes: internal issue (7a518e51908b45c2): 64605561:22202 causes: (33edae1682908f81): many running jobs. project gdfp-xxxx running 10 workflows , project limit active workflows 10. fix this, cancel existing workflow via ui, wait workflow finish or contact dataflow-feedback@google.com request increase in quota. @ com.google.cloud.dataflow.sdk.runners.dataflowpipelinerunner.run(dataflowpipelinerunner.java:221) @ com.google.cloud.dataflow.sdk.runners.blockingdataflowpipelinerunner.run(blockingdataflowpipelinerunner.java:81) @ com.google.cloud.dataflow.sdk.runners.blockingdataflowpipelinerunner.run(blockingdataflowpipelinerunner.java:47) @ com.google.cloud.dataflow.sdk.pipeline.run(pipeline.java:145) @ com.tls.cdf.job.abstractclouddataflowjob.execute(abstractclouddataflowjob.java:100) @ com.tls.cdf.clouddataflowjobexecutor.main(clouddataflowjobexecutor.java:44) caused by: com.google.api.client.googleapis.json.googlejsonresponseexception: 429 many requests { "code" : 429, "errors" : [ { "domain" : "global", "message" : "(40153232ba863405): workflow not created. please try again in few minutes. if still unable create job please contact customer support. causes: (40153232ba8632a6): job not created. please try again in few minutes. if service still isn't working please contact customer support. causes: internal issue (7a518e51908b45c2): 64605561:22202 causes: (33edae1682908f81): many running jobs. project gdfp-xxxx running 10 workflows , project limit active workflows 10. fix this, cancel existing workflow via ui, wait workflow finish or contact dataflow-feedback@google.com request increase in quota.", "reason" : "ratelimitexceeded" } ], "message" : "(40153232ba863405): workflow not created. please try again in few minutes. if still unable create job please contact customer support. causes: (40153232ba8632a6): job not created. please try again in few minutes. if service still isn't working please contact customer support. causes: internal issue (7a518e51908b45c2): 64605561:22202 causes: (33edae1682908f81): many running jobs. project gdfp-xxxx running 10 workflows , project limit active workflows 10. fix this, cancel existing workflow via ui, wait workflow finish or contact dataflow-feedback@google.com request increase in quota.", "status" : "resource_exhausted" } @ com.google.api.client.googleapis.json.googlejsonresponseexception.from(googlejsonresponseexception.java:145) @ com.google.api.client.googleapis.services.json.abstractgooglejsonclientrequest.newexceptiononerror(abstractgooglejsonclientrequest.java:113) @ com.google.api.client.googleapis.services.json.abstractgooglejsonclientrequest.newexceptiononerror(abstractgooglejsonclientrequest.java:40) @ com.google.api.client.googleapis.services.abstractgoogleclientrequest$1.interceptresponse(abstractgoogleclientrequest.java:321) @ com.google.api.client.http.httprequest.execute(httprequest.java:1056) @ com.google.api.client.googleapis.services.abstractgoogleclientrequest.executeunparsed(abstractgoogleclientrequest.java:419) @ com.google.api.client.googleapis.services.abstractgoogleclientrequest.executeunparsed(abstractgoogleclientrequest.java:352) @ com.google.api.client.googleapis.services.abstractgoogleclientrequest.execute(abstractgoogleclientrequest.java:469) @ com.google.cloud.dataflow.sdk.runners.dataflowpipelinerunner.run(dataflowpipelinerunner.java:217) ... 5 more
it's working once again. seems have been problem dataflow service itself.
Comments
Post a Comment