python爬虫scrapy框架爬取网页数据_Python爬虫框架Scrapy学习笔记2爬取Mininova网站种子文件信息...

1. 任务描述

目标网站截图&＃xff1a;

-------------------------------------

可以看到种子文件的列表&＃xff0c;这些链接的url可以用正则表达式表示为&＃xff1a; /tor/\d&＃43;

详情页截图

截图中的&＃xff1a;资源名称, 资源大小&＃xff0c;和资源描述就是我们要抓取的信息。

2. 定义scrapy item.

mininova.pyimport scrapy

class TorrentItem(scrapy.Item):

url &＃61; scrapy.Field()

name &＃61; scrapy.Field()

description &＃61; scrapy.Field()

size &＃61; scrapy.Field()

3. 查看网页源文件&＃xff0c;确定我们要抓取的内容的XPath表达式。对XML Path Language不熟悉的话可以参考&＃xff1a;http://www.w3.org/TR/xpath/ItemXPath

name//hi/text()

size//div[&＃64;id&＃61;&＃39;specifications&＃39;]/p[2]/text()[2]

descripthin//div[&＃64;id&＃61;&＃39;description&＃39;]

4. 最终的代码为&＃xff1a;

mininova.pyimport scrapy

from scrapy.contrib.spiders import CrawlSpider, Rule

from scrapy.contrib.linkextractors import LinkExtractor

class TorrentItem(scrapy.Item):

url &＃61; scrapy.Field()

name &＃61; scrapy.Field()

description &＃61; scrapy.Field()

size &＃61; scrapy.Field()

class MininovaSpider(CrawlSpider):

name &＃61; &＃39;mininova&＃39;

allowed_domains &＃61; [&＃39;mininova.org&＃39;]

start_urls &＃61; [&＃39;http://www.mininova.org/yesterday&＃39;]

rules &＃61; [Rule(LinkExtractor(allow&＃61;[&＃39;/tor/\d&＃43;&＃39;]), &＃39;parse_torrent&＃39;)]

def parse_torrent(self, response):

torrent &＃61; TorrentItem()

torrent[&＃39;url&＃39;] &＃61; response.url

torrent[&＃39;name&＃39;] &＃61; response.xpath("//h1/text()").extract()

torrent[&＃39;description&＃39;] &＃61; response.xpath("//div[&＃64;id&＃61;&＃39;description&＃39;]").extract()

torrent[&＃39;size&＃39;] &＃61; response.xpath("//div[&＃64;id&＃61;&＃39;specifications&＃39;]/p[2]/text()[2]").extract()

return torrent

5. 运行代码

将mininova.py复制到C:\

打开cmd&＃xff0c;输入命令&＃xff1a; scrapy runspider --output&＃61;spider_out.json mininova.py

文件的后缀名很重要&＃xff0c;scrapy会根据后缀名确定输出格式

6. 查看输出