performance - Parallel processing of files in java with ExecutorService does not use all of the CPU power -

-

i have directory contains 1000s of csv files need parse. have implemented executorservice class of java job, wherein assign each thread csv file parse. have 4 cores in machine. efficiency compared single-threaded application. however, when see cpu utilization( using task manager) doesn't seem utilising of cpu power, % of cpu used 30%-40%. wanted know if approach correct.

file dir = new file(file); if(dir.isdirectory()){     file[] files = dir.listfiles();  for(file f : files){     string file_abs_path = f.getabsolutepath();     int index = file_abs_path.lastindexof("/") + 1;     file_name = file_abs_path.substring(index);     futureslist.add(eservice.submit(new myparser(file_abs_path))); }  object gpdocs; for(future<list<myobj>> future:futureslist) { try {     docs = future.get();     arraylist = (list<myobj>)docs;     iterator<myobj> = arraylist.iterator();     while(it.hasnext()){     doc = createdocument(file_name,it.next());     try{         //somefunction(doc);         }catch(exception e){} }}catch (interruptedexception e) {} catch (executionexception e) {} }}

i wondering if approach correct? appreciated.

thanks

the code parser :

public list<myobj> call(){     columnpositionmappingstrategy<myobj> strat =  new columnpositionmappingstrategy<myobj>(); strat.settype(myobj.class); string[] columns = new string[] {//list of columns in csv file};   strat.setcolumnmapping(columns); csvtobean<myobj> csv = new csvtobean<myobj>(); bufferedreader reader = null; string doc_line = ""; string[] docs; string doc = ""; file dir = new file(file_path); try{     int comma_count = 0;     reader = new bufferedreader(new filereader(dir));     while((doc_line = reader.readline()) != null){         docs = doc_line.split(",");     doc += docs[i] + " ";     }     reader.close();     }catch (ioexception e) {/*e.printstacktrace();*/}     return(csv.parse(strat,new stringreader(doc))); }

as commented, task io bound, tasks involving io hard-drive are.

the best performance can hope decouple reading threads processing. probably, single reading thread, reading blocks of data large possible , feeding queue processing yield best overall throughput. number of processing threads whatever necessary keep reading.


Comments

Post a Comment

Popular posts from this blog

ios - Memory not freeing up after popping viewcontroller using ARC -

-
hi project arc based, using uinavigationcontroller make transition between viewcontroller. using profiler analyse happening behind scene memory. noticed when push viewcontroller allocate memory components , when pop not freeing allocated memory. since using arc unable implement dealloc or release component. have analysed in detail , there no memory leak in project. i not using strong property push viewcontroller. here how pushing viewcontroller. viewcontroller *obj = [[viewcontroller alloc] init]; [self.navigationcontroller pushviewcontroller:obj animated:no]; any clue whats going on? should free memory have consumed. please advise your description indicates have retain cycle, leads objects not being deallocated. typical source of retain cycles properties assigned loading nib files (usually declared iboutlet ). two strategies break them , release objects: declare property weak : @property (nonatomic, weak) iboutlet uilabel *statuslabel; set property nil in...
Read more

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more