如何使用BaseItemExporter中的fields_to_export属性来订购我的Scrapy CSV数据？

您现在可以在蜘蛛本身中指定设置. https://doc.scrapy.org/en/latest/topics/settings.html#settings-per-spider

要设置导出的Feed的字段顺序,请设置FEED_EXPORT_FIELDS. https://doc.scrapy.org/en/latest/topics/feed-exports.html#feed-export-fields

下面的蜘蛛转储网站上的所有链接(针对Scrapy 1.4.0编写):

import scrapy
from scrapy.http import HtmlResponse

class DumplinksSpider(scrapy.Spider):
  name = 'dumplinks'
  allowed_domains = ['www.example.com']
  start_urls = ['http://www.example.com/']
  custom_settings = {
    # specifies exported fields and order
    'FEED_EXPORT_FIELDS': ["page", "page_ix", "text", "url"],
  }

  def parse(self, response):
    if not isinstance(response, HtmlResponse):
      return

    a_selectors = response.xpath('//a')
    for i, a_selector in enumerate(a_selectors):
      text = a_selector.xpath('normalize-space(text())').extract_first()
      url = a_selector.xpath('@href').extract_first()
      yield {
        'page_ix': i + 1,
        'page': response.url,
        'text': text,
        'url': url,
      }
      yield response.follow(url, callback=self.parse)  # see allowed_domains

使用此命令运行:

scrapy crawl dumplinks --loglevel=INFO -o links.csv

字段links.csv按照指定的顺序排序FEED_EXPORT_FIELDS.

要使用这样的导出器,您需要创建自己的Item管道来处理您的蜘蛛输出.假设你有简单的情况,并且你希望将所有的蜘蛛输出都放在一个文件中,这就是你应该使用的管道(pipelines.py):

from scrapy import signals
from scrapy.contrib.exporter import CsvItemExporter

class CSVPipeline(object):

  def __init__(self):
    self.files = {}

  @classmethod
  def from_crawler(cls, crawler):
    pipeline = cls()
    crawler.signals.connect(pipeline.spider_opened, signals.spider_opened)
    crawler.signals.connect(pipeline.spider_closed, signals.spider_closed)
    return pipeline

  def spider_opened(self, spider):
    file = open('%s_items.csv' % spider.name, 'w+b')
    self.files[spider] = file
    self.exporter = CsvItemExporter(file)
    self.exporter.fields_to_export = [list with Names of fields to export - order is important]
    self.exporter.start_exporting()

  def spider_closed(self, spider):
    self.exporter.finish_exporting()
    file = self.files.pop(spider)
    file.close()

  def process_item(self, item, spider):
    self.exporter.export_item(item)
    return item

当然,您需要记住在配置文件(settings.py)中添加此管道:

ITEM_PIPELINES = {'myproject.pipelines.CSVPipeline': 300 }