rewrite - Google Cloud Storage - SocketTimeoutException when rewriting a big object to "nearline" bucket -


environment:
java client ("google-api-services-storage", "v1-rev33-1.20.0") using json api (com.google.api.services.storage.storage class).

goal:
move large object "standard" "nearline" bucket using java client (file size 512 mb).

steps:
use "rewrite" api method.

problem:
i'm getting sockettimeoutexception in 20 seconds.

investigation:
same code works fine when use rewrite "standard" bucket "standard" bucket same object.
i've tried apis explorer , created request rewrite object "standard" "nearline" bucket. server responded in 27 seconds , "totalbytesrewritten" property in response half of file size. how , handle such response?
documentation says:
"if source , destination different locations and/or storage classes, rewrite method might require multiple calls."

my code (java):

final storage.objects.rewrite rewriterequest = storage.objects().rewrite(        standard_bucket_name,        source_object_path,        nearline_bucket_name,        target_object_path,        null // no metadata overriding );  rewriterequest.execute(); 

please help.

according documentation, if file split chunks, supposed call rewrite again, using 'rewritetoken' returned in first response rewrite. operation resume, doing 1 more chunk of data. should repeated until response has getdone() == true.

my implementation java api:

private void rewriteuntildone(final string sourcebucket, final string sourcekey,                               final string destbucket,  final string destkey) throws ioexception {     rewriteuntildone(sourcebucket, sourcekey, destbucket, destkey, null); }  private void rewriteuntildone(final string sourcebucket, final string sourcekey,                               final string destbucket, final string destkey,                               @nullable final string rewritetoken)         throws ioexception {      storage.objects.rewrite rewrite = googlestorage.objects().rewrite(sourcebucket, sourcekey, destbucket, destkey, null);     if (rewritetoken != null) {         rewrite.setrewritetoken(rewritetoken);     }     rewriteresponse rewriteresponse = rewrite.execute();      if (!rewriteresponse.getdone()) {         string rewritetoken2 = rewriteresponse.getrewritetoken();         biginteger totalbytesrewritten = rewriteresponse.gettotalbytesrewritten();         log.debug("rewriting not finished, bytes completed: {}. calling rewrite again token {}", totalbytesrewritten, rewritetoken2);         rewriteuntildone(sourcebucket, sourcekey, destbucket, destkey, rewritetoken2);     } } 

edit: also, may have increase read timeout. seems rewrite responds after 27 s default timout 20 s. wrap googlecredentials set read timeout


Comments

Popular posts from this blog

android - MPAndroidChart - How to add Annotations or images to the chart -

javascript - Add class to another page attribute using URL id - Jquery -

firefox - Where is 'webgl.osmesalib' parameter? -