spark udf with data frame -

-

i using spark 1.3. have dataset dates in column (ordering_date column) in yyyy/mm/dd format. want calculations dates , therefore want use jodatime conversions/formatting. here udf have :

 val return_date = udf((str: string, dtf: datetimeformatter) => dtf.formatted(str))

here code udf being called. however, error saying "not applicable". need register udf or missing here?

val user_with_dates_formatted = users.withcolumn(   "formatted_date",   return_date(users("ordering_date"), datetimeformat.forpattern("yyyy/mm/dd") )

i don't believe can pass in datetimeformatter argument udf. can pass in column. 1 solution do:

val return_date = udf((str: string, format: string) => {   datetimeformat.forpatten(format).formatted(str)) })

and then:

val user_with_dates_formatted = users.withcolumn(   "formatted_date",   return_date(users("ordering_date"), lit("yyyy/mm/dd")) )

honestly, though -- both , original algorithms have same problem. both parse yyyy/mm/dd using forpattern every record. better create singleton object wrapped around map[string,datetimeformatter], maybe (thoroughly untested, idea):

object dateformatters {   var formatters = map[string,datetimeformatter]()    def getformatter(format: string) : datetimeformatter = {     if (formatters.get(format).isempty) {       formatters = formatters + (format -> datetimeformat.forpattern(format))     }     formatters.get(format).get   } }

then change udf to:

val return_date = udf((str: string, format: string) => {   dateformatters.getformatter(format).formatted(str)) })

that way, datetimeformat.forpattern(...) called once per format per executor.

one thing note singleton object solution can't define object in spark-shell -- have pack in jar file , use --jars option spark-shell if want use dateformatters object in shell.


Comments

Post a Comment

Popular posts from this blog

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more