Scrapy报400和415错误

今天用Scrapy框架爬取金融APP时出现了一些小问题，折腾了一天，记录一下。

返回415状态码：请求包未加header

首先用Charles抓取手机请求数据包，图如下：

请求包

这是一个POST请求并需要提交表单数据，所以我用了scrapy.FormRequest构造数据包，具体spider代码：

class yilicai(Spider):
name = "yilicai"
urls = "http://api.yilicai.cn/product/all5"
base_url = "https://www.yilicai.cn"
DOWNLOAD_DELAY = 0
count = 0
appmc = "壹理财"
def start_requests(self):
formdata = {
'page': '1',
'sType': '0',
'sTerm': '0',
'sRate': '0',
'sRecover': '0',
'sStart': '0'
}
yield scrapy.FormRequest(self.urls, callback=self.parse, formdata=formdata)
def parse(self,response):
datas = json.loads(response.body)
print(json.dumps(datas, sort_keys=True, indent=2))



运行该爬虫出现415错误：

2018-05-07 17:00:20 [scrapy.core.engine] DEBUG: Crawled (415) (referer: None)
2018-05-07 17:00:20 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <415 http://api.yilicai.cn/product/all5>: HTTP status code is not handled or not allowed
2018-05-07 17:00:21 [scrapy.core.engine] INFO: Closing spider (finished)


去找了一下关于HTTP状态码415的解释：

后来发现是我没有添加header，添加了header的代码修改如下：

headers={
"Accept-Language":"zh-CN,zh;q=0.8",
"User-Agent ":"Mozilla/5.0 (Linux; U; Android 6.0; zh-cn; AOSP on HammerHead Build/MRA58K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30",
"Content-Type":"application/json;charset=utf-8",
"Host":"api.yilicai.cn",
"Accept-Encoding":"gzip",
}
...
...
yield scrapy.FormRequest(self.urls, headers=self.headers,callback=self.parse, formdata=formdata)


返回400状态码：未将提交数据转化为json格式

再次运行415状态码错误算是解决了，但是出现了一个新的错误，报错400：

2018-05-07 17:11:59 [scrapy.core.engine] DEBUG: Crawled (400) (referer: None)
2018-05-07 17:11:59 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <400 http://api.yilicai.cn/product/all5>: HTTP status code is not handled or not allowed
2018-05-07 17:11:59 [scrapy.core.engine] INFO: Closing spider (finished)


真的伤心，就是这个400错误卡了我一整天，我先去找了一下关于400状态码的解释

后来发现是这个请求严格要求提交的表单必须是json格式，所以在提交表单时候需要把formdata转换成json格式，然后进行提交。

由于使用scrapy.FormRequest在构造包时语句formdata=json.dumps(formdata)会报错,所以使用scrapy.Request来进行爬取：

class yilicai(Spider):
name = "yilicai"
urls = "http://api.yilicai.cn/product/all5"
base_url = "https://www.yilicai.cn"
DOWNLOAD_DELAY = 0
count = 0
appmc = "壹理财"
headers={
"Accept-Language":"zh-CN,zh;q=0.8",
"User-Agent ":"Mozilla/5.0 (Linux; U; Android 6.0; zh-cn; AOSP on HammerHead Build/MRA58K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30",
"Content-Type":"application/json;charset=utf-8",
"Host":"api.yilicai.cn",
"Accept-Encoding":"gzip",
}
def start_requests(self):
formdata = {
'page': '1',
'sType': '0',
'sTerm': '0',
'sRate': '0',
'sRecover': '0',
'sStart': '0'
}
temp=json.dumps(formdata)
yield scrapy.Request(self.urls,body=temp,headers=self.headers,callback=self.parse)
def parse(self,response):
datas = json.loads(response.body)
print(json.dumps(datas,sort_keys=True, indent=2))


最后终于成功抓到返回的数据包了，然后愉快的进行数据分析了

2018-05-07 17:47:16 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None)
{
"base_url": "https://www.yilicai.cn",
"current_page": "1",
"new_hand": 1,
"pager": "1",
"pagerParam": {
"count": 16063,
"maxPage": 1607,
"perPage": 10
},
"product_list": [
{
......
}
],
"sid": null,
"status": "0"
}
2018-05-07 17:47:16 [scrapy.core.engine] INFO: Closing spider (finished)
2018-05-07 17:47:16 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
......