我想抓取这个网页(http://www.zhujiage.com.cn/article/List_2.html)和这个网页(http://www.qinbing.cn/jidanjiage/)的url链接地址。
from urllib2 import urlopen from bs4 import BeautifulSoup html = urlopen("http://www.zhujiage.com.cn/article/List_2.html") bsobj = BeautifulSoup(html,'lxml') for link in bsobj.find_all(name="a"): if 'href' in link.attrs: print link.attrs['href']
上面这段代码只改变网址,前一个网址不能抓取网页URL,后面的网址可以抓取。
这个是网站编码的问题, 第一网站编码是gbk, 第二个是utf-8, BeaufifulSoup默认是将网页以uft-8编码, 只需要加个参数from_encoding ="gbk" 就可以解决了
from urllib2 import urlopen from bs4 import BeautifulSoup html = urlopen("http://www.zhujiage.com.cn/article/List_2.html") bsobj = BeautifulSoup(html,'lxml',from_encoding="gbk") for link in bsobj.find_all(name="a"): if 'href' in link.attrs: print link.attrs['href']
也可以
from urllib2 import urlopen from bs4 import BeautifulSoup html = urlopen("http://www.zhujiage.com.cn/article/List_2.html").read().decode('gbk') bsobj = BeautifulSoup(html,'lxml') for link in bsobj.find_all(name="a"): if 'href' in link.attrs: print link.attrs['href']