我scrapy
在python脚本中运行
def setup_crawler(domain): dispatcher.connect(stop_reactor, signal=signals.spider_closed) spider = ArgosSpider(domain=domain) settings = get_project_settings() crawler = Crawler(settings) crawler.configure() crawler.crawl(spider) crawler.start() reactor.run()
它成功运行并停止但结果在哪里?我希望结果采用json格式,我该怎么做?
result = responseInJSON
就像我们使用命令一样
scrapy crawl argos -o result.json -t json
alecxe.. 24
您需要手动设置FEED_FORMAT
和FEED_URI
设置:
settings.overrides['FEED_FORMAT'] = 'json' settings.overrides['FEED_URI'] = 'result.json'
如果要将结果输入变量,可以定义一个Pipeline
将项目收集到列表中的类.使用spider_closed
信号处理程序查看结果:
import json from twisted.internet import reactor from scrapy.crawler import Crawler from scrapy import log, signals from scrapy.utils.project import get_project_settings class MyPipeline(object): def process_item(self, item, spider): results.append(dict(item)) results = [] def spider_closed(spider): print results # set up spider spider = TestSpider(domain='mydomain.org') # set up settings settings = get_project_settings() settings.overrides['ITEM_PIPELINES'] = {'__main__.MyPipeline': 1} # set up crawler crawler = Crawler(settings) crawler.signals.connect(spider_closed, signal=signals.spider_closed) crawler.configure() crawler.crawl(spider) # start crawling crawler.start() log.start() reactor.run()
仅供参考,看看Scrapy如何解析命令行参数.
另请参阅:在Python中的同一进程中捕获stdout.