Convert Spark Java to Spark scala -
i trying convert java code scala in spark, found complicated. possible convert following java code scala? thanks!
javapairrdd<string,tuple2<string,string>> newdatapair = newrecords.maptopair(new pairfunction<string, string, tuple2<string, string>>() { private static final long serialversionuid = 1l; @override public tuple2<string, tuple2<string, string>> call(string t) throws exception { myperson p = (new gson()).fromjson(t, myperson.class); string nameagekey = p.getname() + "_" + p.getage() ; tuple2<string, string> value = new tuple2<string, string>(p.getnationality(), t); tuple2<string, tuple2<string, string>> kvp = new tuple2<string, tuple2<string, string>>(nameagekey.tolowercase(), value); return kvp; } });
i tried following, sure have missed many things. , not clear me how override function in scala ... please suggest or share examples. thank you!
val newdatapair = newrecords.maptopair(new pairfunction<string, string, tuple2<string, string>>() { @override public val call(string t) throws exception { val p = (new gson()).fromjson(t, myperson.class); val nameagekey = p.getname() + "_" + p.getage() ; val value = new tuple2<string, string>(p.getnationality(), t); val kvp = new tuple2<string, tuple2<string, string>>(nameagekey.tolowercase(), value); return kvp; } });
literal translations spark-java spark-scala typically don't work because spark-java introduces many artifacts cope limited type system in java. examples in case: maptopair
in java map
in scala. tuple2
has more terse syntax (a,b)
applying (and more) snippet:
val newdatapair = newrecords.map{t => val p = (new gson()).fromjson(t, classof[myperson]) val nameagekey = p.getname + "_" + p.getage val value = (p.getnationality(), t) (nameagekey.tolowercase(), value) }
it made bit more concise wanted keep same structure java counterpart facilitate understanding of it.
Comments
Post a Comment