python - 爬虫爬https站点怎么处理的?

 陈民翔芷昌柏淑 发布于 2022-10-30 16:50

一段模拟登入知乎的问题
PS:这里不讨论验证码问题,我的账号登入没有提示验证码

代码信息:

# /usr/bin/python
#coding:utf-8
__author__ = 'eyu Fanne'

import requests,time
from bs4 import BeautifulSoup

begintime = time.time()



headers={
    "Host":"www.zhihu.com",
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36",
    "Accept":"*/*",
    "Accept-Language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3",
    "Accept-Encoding":"gzip, deflate",
    "Content-Type":"application/x-www-form-urlencoded; charset=UTF-8",
    "X-Requested-With":"XMLHttpRequest",
    "Connection":"keep-alive"
}


login_url=r'https://www.zhihu.com/login/email'

html_txt = requests.get(login_url,verify=False).text
html_soup = BeautifulSoup(html_txt,'lxml')
xsrf = html_soup.find("input",{"name":"_xsrf"})['value']

#print xsrf
#print html_txt
s = requests.session()

url_data={
    "_xsrf":xsrf,
    "email":"******",
    "password":"********",
    'remember_me':'true'
}

s_login=s.post(login_url,data=url_data,headers=headers,verify=False)
print s_login.text

endtime = time.time()
usetime =  endtime - begintime
print "执行脚本总用时 %s 秒" % usetime


#print s.get('https://www.zhihu.com',verify=False).text

执行结果:

C:\Python27_x86\lib\site-packages\requests\packages\urllib3\connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
C:\Python27_x86\lib\site-packages\requests\packages\urllib3\connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
C:\Python27_x86\lib\site-packages\requests\packages\urllib3\connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
{"r":0,
 "msg": "\u767b\u9646\u6210\u529f"
}
执行脚本总用时 73.2650001049 秒

在模拟登入的时候,不知道是不是因为https的问题处理的好慢,整个模拟登入要73s才完成,不知卡在哪里了。
像类似这种问题的应该怎么处理。

2 个回答
  • 可以用cProfile模块,先把所有过程封装到一个函数里,然后cProfile看看哪个函数调用的时间最长。

    不过你这程序在我这里运行起来挺快的,应该和你的网络环境有关。

    2022-10-31 21:48 回答
  • from requests.packages.urllib3.exceptions import InsecureRequestWarning,InsecurePlatformWarning
    requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
    requests.packages.urllib3.disable_warnings(InsecurePlatformWarning)

    可以解决告警问题,以及解决速度问题

    2022-10-31 21:48 回答
撰写答案
今天,你开发时遇到什么问题呢?
立即提问
热门标签
PHP1.CN | 中国最专业的PHP中文社区 | PNG素材下载 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有