How to capture job status with python multiprocessing.Process -

-

(using python 3.4)

this question identifying completion of jobs ran using multiprocessing.process , send end result of job function further processing.

i'm using multiprocessing.process run 6 unix jobs in parallel. 6 jobs submitted using shell script ./runme runme simple shell script having command line calling tool. 

#!/bin/csh nc run -r cpus/4 -- qrc -cmd designname.ccl

nc bsub

end result of run creates text file consumed post processing script.

my existing python code has following logic:

def sub_runqrc (designname, outputdir):     qrcrun_process = subprocess.popen("./"+runme, shell+true, cwd=designdir, stdout=subprocess.pipe, stderr=subprocess.pipe)     line in qrcrun_process.stdout:          ### grep job id ###          return_dict.update({designname:<jobid>})     return (return_dict)   #main function logic line in designlist:     #### designlist file , has 6 names in ####     #### other logic #####     qrc_processes = multiprocessing.process(target=sub_runqrc, args=(line, testcase_dir))     qrc_jobs.append(qrc_processes)     qrc_processes.start() proc in qrc_jobs:     proc.join()

from above logic each process done job id collected. tool continuing work on farm. how know if status of job done? , call function start post processing output text file available.

i'm using periodic function every 15mins check status of job using command similar bjobs (on "nc" nc info jobid) job info unavailable on server job finished.

what best way find out if job finished? there way can capture exit code kind of thing variable in main function , call pass designname function start post processing output file?

def sub_postprocess (designname, output.txt, outputdir):     ### run post process ####      return ("post processing done "+designname)

is using multiprocessing.process correct way or these kind of jobs has submitted in other ways?

note: started writing code in python 15days ago. struggling understand classes , multiprocesses. suggestions/reccommedations?

in case if question not clear please let me know.


Comments

Post a Comment

Popular posts from this blog

Django REST Framework perform_create: You cannot call `.save()` after accessing `serializer.data` -

-
i have following viewset: class artistprofileviewset(viewsets.modelviewset): queryset = artistprofile.objects.all() serializer_class = artistprofileserializer def perform_create(self, serializer): if serializer.is_valid(): serializer.save() with following serializers: class simpleartisttrackserializer(serializers.hyperlinkedmodelserializer): class meta: model = artisttrack fields = (...my fields...) class artistprofileserializer(serializers.hyperlinkedmodelserializer): owners = userserializer(many=true, required=false) tracks = simpleartisttrackserializer(many=true, required=false) class meta: model = artistprofile fields = (...my fields...) i getting following error: assertionerror: cannot call `.save()` after accessing `serializer.data`.if need access data before committing database inspect 'serializer.validated_data' instead. i don't see editing serializer.data object. th...
Read more

Why does Go error when trying to marshal this JSON? -

-
i'm trying rather simple example of trying decode json file structs using golang. however, when attempting error that json: cannot unmarshal object go value of type []main.track which don't understand. here's code in question, , json can found here . package main import ( "encoding/json" "fmt" "net/http" "os" ) // tracks group of track types type tracks struct { tracks []track } // track represents singular track type track struct { name string url string artist artist } // artist represents single artist type artist struct { name string mbid string url string } func main() { url := "http://ws.audioscrobbler.com/2.0/?method=geo.gettoptracks&api_key=c1572082105bd40d247836b5c1819623&format=json&country=netherlands" response, err := http.get(url) if err != nil { fmt.printf("%s\n", err) os.exit(1) } de...
Read more