我正在尝试使用Python多处理模块生成多个并行进程.基本上,我做了类似的事情
pool = Pool(30) results = [pool.apply_async(foo, (trainData, featureVector, terms, selLabel)) for selLabel in selLabels] for r in results: tmp = r.get() modelFiles[tmp[0]] = tmp[1]
产生了30个进程,但是,似乎大多数进程已进入休眠状态,而实际只有一个进程正在运行.以下是我从ps得到的:
PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 31267 74.6 2.4 7125412 6360080 pts/1 Sl+ 13:06 24:25 \_ python2.6 /home/PerlModules/Python/DoOVA.py 31427 27.4 2.3 6528532 6120904 pts/1 R+ 13:20 5:18 \_ python2.6 /home/PerlModules/Python/DoOVA.py 31428 0.0 1.3 4024724 3617016 pts/1 S+ 13:20 0:00 \_ python2.6 /home/PerlModules/Python/DoOVA.py 31429 0.0 1.3 4024724 3617016 pts/1 S+ 13:20 0:00 \_ python2.6 /home/PerlModules/Python/DoOVA.py 31430 0.0 1.3 4024724 3617016 pts/1 S+ 13:20 0:00 \_ python2.6 /home/PerlModules/Python/DoOVA.py
DoOVA.py
是我正在运行的脚本.他们中的大多数都有一个地位S+
.
谁能给我一些关于问题的线索?我知道输入争论的featureVector
规模相当大,比如大约300MB.那会是个问题吗?我运行的机器有几TB的内存.
foo做的事情如下:
def foo(trainData, featureVector, terms, selLabel, penalty): outputFile = 'train_'+selLabel+'.dat' annotation = dict() for id in trainData: if trainData[id] == selLabel: annotation[id] = '1' else: annotation[id] = '-1' try: os.mkdir(selLabel) os.chdir(selLabel) except OSError: os.chdir(selLabel) ###Some more functions, which involves a command line call through call from subprocess module os.chdir('../') return (selLabel, 'SVM_' + selLabel + '.model')
所有其他输入参数的大小都很小.机器至少有100 cpus.在每次运行中,即使在创建任何目录之前,脚本也需要很长时间,尽管在os.mkdir()之前foo中没有发生重大计算.