How to capture job status with python multiprocessing.Process -


(using python 3.4)

this question identifying completion of jobs ran using multiprocessing.process , send end result of job function further processing.

i'm using multiprocessing.process run 6 unix jobs in parallel. 6 jobs submitted using shell script ./runme runme simple shell script having command line calling tool.

#!/bin/csh nc run -r cpus/4 -- qrc -cmd designname.ccl  

nc bsub

end result of run creates text file consumed post processing script.

my existing python code has following logic:

def sub_runqrc (designname, outputdir):     qrcrun_process = subprocess.popen("./"+runme, shell+true, cwd=designdir, stdout=subprocess.pipe, stderr=subprocess.pipe)     line in qrcrun_process.stdout:          ### grep job id ###          return_dict.update({designname:<jobid>})     return (return_dict)   #main function logic line in designlist:     #### designlist file , has 6 names in ####     #### other logic #####     qrc_processes = multiprocessing.process(target=sub_runqrc, args=(line, testcase_dir))     qrc_jobs.append(qrc_processes)     qrc_processes.start() proc in qrc_jobs:     proc.join() 

from above logic each process done job id collected. tool continuing work on farm. how know if status of job done? , call function start post processing output text file available.

i'm using periodic function every 15mins check status of job using command similar bjobs (on "nc" nc info jobid) job info unavailable on server job finished.

what best way find out if job finished? there way can capture exit code kind of thing variable in main function , call pass designname function start post processing output file?

def sub_postprocess (designname, output.txt, outputdir):     ### run post process ####      return ("post processing done "+designname) 

is using multiprocessing.process correct way or these kind of jobs has submitted in other ways?

note: started writing code in python 15days ago. struggling understand classes , multiprocesses. suggestions/reccommedations?

in case if question not clear please let me know.


Comments

Popular posts from this blog

html - Styling progress bar with inline style -

java - Oracle Sql developer error: could not install some modules -

How to use autoclose brackets in Jupyter notebook? -