java - How to output files with a specific extension (like .csv) in Hadoop, using MultipleOutputs class -

-

i have mapreduce program uses multipleoutputsto output result multiple files. reducer looks this: 

private multipleoutputs mo = new multipleoutputs<nullwritable, text>(context); ... public void reduce(edge keys, iterable<nullwritable> values, context context)             throws ioexception, interruptedexception {         string date = records.formatdate(millis);         out.set(keys.get(0) + "\t" + keys.get(1));         parser.parse(key);          string filepath = string.format("%s/part", parser.getfileid());         mo.write(noval, out, filepath);     }

this similar example in book hadoop: definitive guide - however, problem outputs files plain text. want files outputted .csv files , haven't managed find explanation on in book or online.

how can done?

have tried iterate through output folder after completion of job object in driver rename files?

as long emit in reducer (the text should line in csv values separated semicolon or whatever need) can give try this:

job job = new job(getconf()); //... //your job setup, including output config  job.setoutputkeyclass(text.class); job.setoutputvalueclass(nullwritable.class); //... boolean success = job.waitforcompletion(true); if (success){     filesystem hdfs = filesystem.get(getconf());     filestatus fs[] = hdfs.liststatus(new path(outputpath));     if (fs != null){          (filestatus afile : fs) {             if (!afile.isdir()) {                 hdfs.rename(afile.getpath(), new path(afile.getpath().tostring()+".csv"));             }         }     } }

Comments

Post a Comment

Popular posts from this blog

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more