rewrite - Google Cloud Storage - SocketTimeoutException when rewriting a big object to "nearline" bucket -
environment:
java client ("google-api-services-storage", "v1-rev33-1.20.0") using json api (com.google.api.services.storage.storage class).
goal:
move large object "standard" "nearline" bucket using java client (file size 512 mb).
steps:
use "rewrite" api method.
problem:
i'm getting sockettimeoutexception in 20 seconds.
investigation:
same code works fine when use rewrite "standard" bucket "standard" bucket same object.
i've tried apis explorer , created request rewrite object "standard" "nearline" bucket. server responded in 27 seconds , "totalbytesrewritten" property in response half of file size. how , handle such response?
documentation says:
"if source , destination different locations and/or storage classes, rewrite method might require multiple calls."
my code (java):
final storage.objects.rewrite rewriterequest = storage.objects().rewrite( standard_bucket_name, source_object_path, nearline_bucket_name, target_object_path, null // no metadata overriding ); rewriterequest.execute();
please help.
according documentation, if file split chunks, supposed call rewrite again, using 'rewritetoken' returned in first response rewrite. operation resume, doing 1 more chunk of data. should repeated until response has getdone() == true.
my implementation java api:
private void rewriteuntildone(final string sourcebucket, final string sourcekey, final string destbucket, final string destkey) throws ioexception { rewriteuntildone(sourcebucket, sourcekey, destbucket, destkey, null); } private void rewriteuntildone(final string sourcebucket, final string sourcekey, final string destbucket, final string destkey, @nullable final string rewritetoken) throws ioexception { storage.objects.rewrite rewrite = googlestorage.objects().rewrite(sourcebucket, sourcekey, destbucket, destkey, null); if (rewritetoken != null) { rewrite.setrewritetoken(rewritetoken); } rewriteresponse rewriteresponse = rewrite.execute(); if (!rewriteresponse.getdone()) { string rewritetoken2 = rewriteresponse.getrewritetoken(); biginteger totalbytesrewritten = rewriteresponse.gettotalbytesrewritten(); log.debug("rewriting not finished, bytes completed: {}. calling rewrite again token {}", totalbytesrewritten, rewritetoken2); rewriteuntildone(sourcebucket, sourcekey, destbucket, destkey, rewritetoken2); } }
edit: also, may have increase read timeout. seems rewrite responds after 27 s default timout 20 s. wrap googlecredentials set read timeout
Comments
Post a Comment