Scrapy重试或重定向中间件

我今天遇到了同样的问题,一个使用301..303重定向的网站,但有时也有元重定向.我已经构建了一个重试中间件,并使用了重定向中间件的一些块:

from scrapy.contrib.downloadermiddleware.retry import RetryMiddleware
from scrapy.selector import HtmlXPathSelector
from scrapy.utils.response import get_meta_refresh
from scrapy import log

class CustomRetryMiddleware(RetryMiddleware):

    def process_response(self, request, response, spider):
        url = response.url
        if response.status in [301, 307]:
            log.msg("trying to redirect us: %s" %url, level=log.INFO)
            reason = 'redirect %d' %response.status
            return self._retry(request, reason, spider) or response
        interval, redirect_url = get_meta_refresh(response)
        # handle meta redirect
        if redirect_url:
            log.msg("trying to redirect us: %s" %url, level=log.INFO)
            reason = 'meta'
            return self._retry(request, reason, spider) or response
        hxs = HtmlXPathSelector(response)
        # test for captcha page
        captcha = hxs.select(".//input[contains(@id, 'captchacharacters')]").extract()
        if captcha:
            log.msg("captcha page %s" %url, level=log.INFO)
            reason = 'capcha'
            return self._retry(request, reason, spider) or response
        return response

为了使用这个中间件,最好在settings.py中禁用此项目的现有重定向中间件:

DOWNLOADER_MIDDLEWARES = {
                         'YOUR_PROJECT.scraper.middlewares.CustomRetryMiddleware': 120,
                          'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware': None,
                          'scrapy.contrib.downloadermiddleware.redirect.MetaRefreshMiddleware': None,
}