我正在尝试使用Python执行一些昂贵的科学计算.我必须读取存储在csv文件中的一堆数据然后进行处理.由于每个过程需要很长时间,我有8个处理器可供使用,我试图使用该Pool
方法Multiprocessing
.
这就是我构建多处理调用的方式:
pool = Pool() vector_components = [] for sample in range(samples): vector_field_x_i = vector_field_samples_x[sample] vector_field_y_i = vector_field_samples_y[sample] vector_component = pool.apply_async(vector_field_decomposer, args=(x_dim, y_dim, x_steps, y_steps, vector_field_x_i, vector_field_y_i)) vector_components.append(vector_component) pool.close() pool.join() vector_components = map(lambda k: k.get(), vector_components) for vector_component in vector_components: CsvH.write_vector_field(vector_component, '../CSV/RotationalFree/rotational_free_x_'+str(sample)+'.csv')
我正在运行500个样本的数据集,大小等于100(x_dim)乘以100(y_dim).
在那之前一切正常.
然后我收到500个400 x 400样本的数据集.
运行时,我在调用时遇到错误get
.
我还尝试运行400 x 400的单个样本并得到相同的错误.
Traceback (most recent call last): File "__init__.py", line 33, inVfD.samples_vector_field_decomposer(samples, x_dim, y_dim, x_steps, y_steps, vector_field_samples_x, vector_field_samples_y) File "/export/home/pceccon/VectorFieldDecomposer/Sources/Controllers/VectorFieldDecomposerController.py", line 43, in samples_vector_field_decomposer vector_components = map(lambda k: k.get(), vector_components) File "/export/home/pceccon/VectorFieldDecomposer/Sources/Controllers/VectorFieldDecomposerController.py", line 43, in vector_components = map(lambda k: k.get(), vector_components) File "/export/home/pceccon/.pyenv/versions/2.7.5/lib/python2.7/multiprocessing/pool.py", line 554, in get raise self._value MemoryError
我该怎么办?
先感谢您.