Convert Spark Java to Spark scala -


i trying convert java code scala in spark, found complicated. possible convert following java code scala? thanks!


javapairrdd<string,tuple2<string,string>> newdatapair = newrecords.maptopair(new pairfunction<string, string, tuple2<string, string>>() {              private static final long serialversionuid = 1l;              @override             public tuple2<string, tuple2<string, string>> call(string t) throws exception {                 myperson p = (new gson()).fromjson(t, myperson.class);                  string nameagekey = p.getname() + "_" + p.getage() ;                  tuple2<string, string> value = new tuple2<string, string>(p.getnationality(), t);                 tuple2<string, tuple2<string, string>> kvp =                     new tuple2<string, tuple2<string, string>>(nameagekey.tolowercase(), value);                 return kvp;             }         }); 

i tried following, sure have missed many things. , not clear me how override function in scala ... please suggest or share examples. thank you!

val newdatapair = newrecords.maptopair(new pairfunction<string, string, tuple2<string, string>>() {          @override         public val call(string t) throws exception {             val p = (new gson()).fromjson(t, myperson.class);             val nameagekey = p.getname() + "_" + p.getage() ;             val value = new tuple2<string, string>(p.getnationality(), t);             val kvp =                 new tuple2<string, tuple2<string, string>>(nameagekey.tolowercase(), value);             return kvp;         }     }); 

literal translations spark-java spark-scala typically don't work because spark-java introduces many artifacts cope limited type system in java. examples in case: maptopair in java map in scala. tuple2 has more terse syntax (a,b)

applying (and more) snippet:

val newdatapair = newrecords.map{t =>                     val p = (new gson()).fromjson(t, classof[myperson])     val nameagekey = p.getname + "_" + p.getage     val value = (p.getnationality(), t)     (nameagekey.tolowercase(), value) } 

it made bit more concise wanted keep same structure java counterpart facilitate understanding of it.


Comments

Popular posts from this blog

android - MPAndroidChart - How to add Annotations or images to the chart -

javascript - Add class to another page attribute using URL id - Jquery -

firefox - Where is 'webgl.osmesalib' parameter? -