python文件分割合并算法_python学习——大文件分割与合并

在平常的生活中&＃xff0c;我们会遇到下面这样的情况&＃xff1a;

你下载了一个比较大型的游戏(假设有10G)&＃xff0c;现在想跟你的同学一起玩&＃xff0c;你需要把这个游戏拷贝给他。

然后现在有一个问题是文件太大(我们不考虑你有移动硬盘什么的情况)&＃xff0c;假设现在只有一个2G或4G的优盘&＃xff0c;该怎么办呢&＃xff1f;

有很多方法&＃xff0c;例如winrar压缩的时候分成很多小卷&＃xff0c;这里不累述。

在学习python之后&＃xff0c;我们自己就可以解决这个问题啦。

我们可以自己写一个脚本去分割合并文件&＃xff0c;将文件分割成适合优盘大小的小文件&＃xff0c;在拷贝&＃xff0c;然后再合并。

import sys,os

kilobytes&＃61; 1024megabytes&＃61; kilobytes*1000chunksize&＃61; int(200*megabytes)#defaultchunksize

def split(fromfile,todir,chunksize&＃61;chunksize):ifnot os.path.exists(todir):#check whether todir exists or not

os.mkdir(todir)else:for fname inos.listdir(todir):

os.remove(os.path.join(todir,fname))

partnum&＃61; 0inputfile&＃61; open(fromfile,&＃39;rb&＃39;)#open the fromfilewhileTrue:

chunk&＃61;inputfile.read(chunksize)if not chunk: #check the chunk isemptybreakpartnum&＃43;&＃61; 1filename&＃61; os.path.join(todir,(&＃39;part%04d&＃39;%partnum))

fileobj&＃61; open(filename,&＃39;wb&＃39;)#make partfile

fileobj.write(chunk) #write data into partfile

fileobj.close()returnpartnumif __name__&＃61;&＃61;&＃39;__main__&＃39;:

fromfile&＃61; input(&＃39;File to be split?&＃39;)

todir&＃61; input(&＃39;Directory to store part files?&＃39;)

chunksize&＃61; int(input(&＃39;Chunksize to be split?&＃39;))

absfrom,absto&＃61;map(os.path.abspath,[fromfile,todir])

print(&＃39;Splitting&＃39;,absfrom,&＃39;to&＃39;,absto,&＃39;by&＃39;,chunksize)try:

parts&＃61;split(fromfile,todir,chunksize)

except:

print(&＃39;Error during split:&＃39;)

print(sys.exc_info()[0],sys.exc_info()[1])else:

print(&＃39;split finished:&＃39;,parts,&＃39;parts are in&＃39;,absto)

下面是脚本运行的例子&＃xff1a;

我们在F有一个X—MEN1.rar文件&＃xff0c;1.26G大小&＃xff0c;我们现在把它分割成400000000bit(大约380M)的文件。

Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64bit (AMD64)] on win32

Type"copyright", "credits" or "license()" formore information.>>> &＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61; RESTART &＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;

>>>File to be split?F:\X-MEN1.rar

Directory to store part files?F:\split

Chunksize to be split?400000000Splitting F:\X-MEN1.rar to F:\split by 400000000split finished:4 parts are inF:\split>>>

这是分割后的文件&＃xff1a;

下面是文件合并脚本&＃xff1a;

import sys,os

def joinfile(fromdir,filename,todir):ifnot os.path.exists(todir):

os.mkdir(todir)ifnot os.path.exists(fromdir):

print(&＃39;Wrong directory&＃39;)

outfile&＃61; open(os.path.join(todir,filename),&＃39;wb&＃39;)

files&＃61; os.listdir(fromdir) #list all the part files inthe directory

files.sort() #sort part files to readinorderfor file infiles:

filepath&＃61;os.path.join(fromdir,file)

infile&＃61; open(filepath,&＃39;rb&＃39;)

data&＃61;infile.read()

outfile.write(data)

infile.close()

outfile.close()if __name__&＃61;&＃61;&＃39;__main__&＃39;:

fromdir&＃61; input(&＃39;Directory containing part files?&＃39;)

filename&＃61; input(&＃39;Name of file to be recreated?&＃39;)

todir&＃61; input(&＃39;Directory to store recreated file?&＃39;)try:

joinfile(fromdir,filename,todir)

except:

print(&＃39;Error joining files:&＃39;)

print(sys.exc_info()[0],sys.exc_info()[1])

运行合并脚本&＃xff0c;将上面分割脚本分割的文件重组&＃xff1a;

Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 10:45:13) [MSC v.1600 64bit (AMD64)] on win32

Type"copyright", "credits" or "license()" formore information.>>> &＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61; RESTART &＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;

>>>Directory containing part files?F:\split

Name of file to be recreated?xman1.rar

Directory to store recreated file?F:\>>>

运行之后可以看到F盘下生成了重组的xman.rar