python爬虫之豆瓣音乐top250

回家很久了&＃xff0c;实在熬不住&＃xff0c;想起来爬点数据玩一玩&＃xff0c;之前自己笔记本是win7加ubuntu16.04双系统&＃xff0c;本打算在ubuntu里写代码的&＃xff0c;可是回到家ubuntu打开一直是紫屏&＃xff0c;百度知乎方法用了也没解决&＃xff0c;厉害的兄弟可以教下我哦&＃xff0c;过年有红包哦&＃xff01;&＃xff01;然后就还是在win7下开始写代码了&＃xff08;电脑太卡&＃xff0c;一直不想装Python&＃xff09;&＃xff0c;今天爬的是豆瓣音乐top250&＃xff0c;比较简单&＃xff0c;主要是练练手。

import requests
import re
from bs4 import BeautifulSoup
import time
import pymongoclient &＃61; pymongo.MongoClient(&＃39;localhost&＃39;, 27017)
douban &＃61; client[&＃39;douban&＃39;]
musictop &＃61; douban[&＃39;musictop&＃39;]headers &＃61; {&＃39;User-Agent&＃39;:&＃39;Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36&＃39;
}
urls &＃61; [&＃39;https://music.douban.com/top250?start&＃61;{}&＃39;.format(str(i)) for i in range(0,250,25)]def get_url_music(url):wb_data &＃61; requests.get(url,headers&＃61;headers)soup &＃61; BeautifulSoup(wb_data.text,&＃39;lxml&＃39;)music_hrefs &＃61; soup.select(&＃39;a.nbg&＃39;)for music_href in music_hrefs:get_music_info(music_href[&＃39;href&＃39;])time.sleep(2)def get_music_info(url):wb_data &＃61; requests.get(url,headers&＃61;headers)soup &＃61; BeautifulSoup(wb_data.text,&＃39;lxml&＃39;)names &＃61; soup.select(&＃39;h1 > span&＃39;)authors &＃61; soup.select(&＃39;span.pl > a&＃39;)styles &＃61; re.findall(&＃39;流派: (.*?)
&＃39;,wb_data.text,re.S)times &＃61; re.findall(&＃39;发行时间: (.*?)
&＃39;,wb_data.text,re.S)contents &＃61; soup.select(&＃39;span.short > span&＃39;)if len(names) &＃61;&＃61; 0:name &＃61; &＃39;缺失&＃39;else:name &＃61; names[0].get_text()if len(authors) &＃61;&＃61; 0:author &＃61; &＃39;佚名&＃39;else:author &＃61; authors[0].get_text()if len(styles) &＃61;&＃61; 0:style &＃61; &＃39;未知&＃39;else:style &＃61; styles[0].split(&＃39;\n&＃39;)[0]if len(times) &＃61;&＃61; 0:time &＃61; &＃39;未知&＃39;else:time &＃61; times[0].split(&＃39;-&＃39;)[0]if len(contents) &＃61;&＃61; 0:content &＃61; &＃39;无&＃39;else:content &＃61; contents[0].get_text()info &＃61; {&＃39;name&＃39;:name,&＃39;author&＃39;:author,&＃39;style&＃39;:style,&＃39;time&＃39;:time,&＃39;content&＃39;:content}musictop.insert_one(info)for url in urls:get_url_music(url)


1加了请求头&＃xff08;本来没加&＃xff0c;调试几次突然没数据了&＃xff0c;加了请求头开始也没好&＃xff0c;后来又好了&＃xff0c;可能是网络原因&＃xff09;
2这次是进入信息页爬的数据&＃xff08;上次爬电影没采用这种方法&＃xff0c;缺少了部分数据&＃xff09;
3数据的预处理用了很多if函数&＃xff0c;厉害的兄弟有什么优化的方法。