如何在不制作副本的情况下缩小pandasDataFrame的尺寸？-howtodownsizeapandasDataFramewithoutmakingacopy?

0  

I feel your pain. Pandas is not a memory-friendly library, and 500Mb can quickly turn into >16Gb and shredding performance.

我感觉到你的痛苦。 Pandas不是一个对内存友好的库，500Mb可以很快变成> 16Gb并破坏性能。

However, one thing that's worked for me is memmap. You can use memmap to page in numpy arrays and matrices just about as fast as your databus permits. And as an added benefit, unused pages may be unloaded.

然而，有一件事对我有用的是memmap。您可以使用memmap在numpy数组和矩阵中进行分页，其速度与数据总线允许的速度一样快。并且作为额外的好处，可以卸载未使用的页面。

See here for details. With some work, these memmap np arrays can be used to back a pd.Series or a pd.DataFrame without copying. However, you may find that Pandas later copies you data as you proceed. So, my advice: create a memmap file, and stay in numpy-land.

详情请见此处。通过一些工作，这些memmap np数组可用于支持pd.Series或pd.DataFrame而无需复制。但是，您可能会发现Pandas稍后会在您继续时复制数据。所以，我的建议是：创建一个memmap文件，并保持在numpy-land。

Your other alternative is to use HDFS.

您的另一种选择是使用HDFS。