目标是从互联网上下载文件,并从中创建一个文件对象,或者像对象这样的文件,而不必触摸硬盘.这只是为了我的知识,想要知道它是否可行或实用,特别是因为我想看看我是否可以避免编写文件删除行.
这就是我通常从网上下载内容并将其映射到内存的方式:
import requests import mmap u = requests.get("http://www.pythonchallenge.com/pc/def/channel.zip") with open("channel.zip", "wb") as f: # I want to eliminate this, as this writes to disk f.write(u.content) with open("channel.zip", "r+b") as f: # and his as well, because it reads from disk mm = mmap.mmap(f.fileno(), 0) mm.seek(0) print mm.readline() mm.close() # question: if I do not include this, does this become a memory leak?
jfs.. 28
r.raw
(HTTPResponse
)已经是一个类文件对象(只是传递stream=True
):
#!/usr/bin/env python import sys import requests # $ pip install requests from PIL import Image # $ pip install pillow url = sys.argv[1] r = requests.get(url, stream=True) r.raw.decode_content = True # Content-Encoding im = Image.open(r.raw) #NOTE: it requires pillow 2.8+ print(im.format, im.mode, im.size)
通常,如果你有一个bytestring; 您可以将其包装为f = io.BytesIO(r.content)
,以获取类似文件的对象而不触及磁盘:
#!/usr/bin/env python import io import zipfile from contextlib import closing import requests # $ pip install requests r = requests.get("http://www.pythonchallenge.com/pc/def/channel.zip") with closing(r), zipfile.ZipFile(io.BytesIO(r.content)) as archive: print({member.filename: archive.read(member) for member in archive.infolist()})
您无法直接传递r.raw
,ZipFile()
因为前者是不可搜索的文件.
我想看看我是否可以规避必须编写文件删除行的代码
tempfile
可以自动删除文件f = tempfile.SpooledTemporaryFile(); f.write(u.content)
.直到.fileno()
调用方法(如果某些api需要真实文件)或maxsize
到达方法; 数据保存在内存中.即使数据写在磁盘上; 文件一关闭就会被删除.
r.raw
(HTTPResponse
)已经是一个类文件对象(只是传递stream=True
):
#!/usr/bin/env python import sys import requests # $ pip install requests from PIL import Image # $ pip install pillow url = sys.argv[1] r = requests.get(url, stream=True) r.raw.decode_content = True # Content-Encoding im = Image.open(r.raw) #NOTE: it requires pillow 2.8+ print(im.format, im.mode, im.size)
通常,如果你有一个bytestring; 您可以将其包装为f = io.BytesIO(r.content)
,以获取类似文件的对象而不触及磁盘:
#!/usr/bin/env python import io import zipfile from contextlib import closing import requests # $ pip install requests r = requests.get("http://www.pythonchallenge.com/pc/def/channel.zip") with closing(r), zipfile.ZipFile(io.BytesIO(r.content)) as archive: print({member.filename: archive.read(member) for member in archive.infolist()})
您无法直接传递r.raw
,ZipFile()
因为前者是不可搜索的文件.
我想看看我是否可以规避必须编写文件删除行的代码
tempfile
可以自动删除文件f = tempfile.SpooledTemporaryFile(); f.write(u.content)
.直到.fileno()
调用方法(如果某些api需要真实文件)或maxsize
到达方法; 数据保存在内存中.即使数据写在磁盘上; 文件一关闭就会被删除.
这就是我最终做的事情.
import zipfile import requests import StringIO u = requests.get("http://www.pythonchallenge.com/pc/def/channel.zip") f = StringIO.StringIO() f.write(u.content) def extract_zip(input_zip): input_zip = zipfile.ZipFile(input_zip) return {i: input_zip.read(i) for i in input_zip.namelist()} extracted = extract_zip(f)
你的答案是u.content
.内容是在内存中.除非您将其写入文件,否则它不会存储在磁盘上.