Python案例篇：某短视频采集（不用等着大数据给你推送了）

前言

采集目标

工具准备

开发工具&＃xff1a;pycharm
开发环境&＃xff1a;python3.7&＃xff0c; Windows10
使用工具包&＃xff1a;requests

项目思路解析

我们今天需要获取的是某短视频的点赞视频信息&＃xff0c;首先定位到我们需要的数据位置&＃xff0c;通过抓包的方式进行准确的数据&＃xff0c;点击快手页面进行登录&＃xff0c;点击个人中心&＃xff0c;点击我赞的视频&＃xff0c;在当前页面进行抓包&＃xff0c;获取到我们想要的数据信息

明确到自己需要采集的数据之后做我们爬虫的第一步&＃xff0c;获取到我们目标地址&＃xff1a;某短视频首页&＃xff0c;通过requests发送网络请求&＃xff0c;带上我们需要的请求头&＃xff0c;这个请求头需要注意&＃xff0c;他的请求头需要带上accept: /&＃xff0c;不然在获取下一页的时候页数是不会变化的&＃xff0c;这个需要重点注意&＃xff0c;请求头入下&＃xff1a;

headers &＃61; {&＃39;accept&＃39;: &＃39;*/*&＃39;,&＃39;content-type&＃39;: &＃39;application/json&＃39;,&＃39;COOKIE&＃39;: &＃39;&＃39;,&＃39;Host&＃39;: &＃39;www.kuaishou.com&＃39;,&＃39;Origin&＃39;: &＃39;https://www.xxxxxxxxxx.com&＃39;,&＃39;Referer&＃39;: &＃39;https://www.xxxxxxxxx.com/profile/3x382umv98zjz79&＃39;,&＃39;User-Agent&＃39;: &＃39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36&＃39;,
}


发送的是post请求需要携带的参数&＃xff0c;pcursor是用来控制页数信息的,传递的数据是载荷数据&＃xff0c;传输时以json接受就行&＃xff0c;一下是提交的数据信息&＃xff1a;

data &＃61; {"operationName": "visionProfileLikePhotoList","variables": {"pcursor": &＃39;&＃39;,"page": "profile"},"query": "fragment photoContent on PhotoEntity {\n id\n duration\n caption\n likeCount\n viewCount\n realLikeCount\n coverUrl\n photoUrl\n photoH265Url\n manifest\n manifestH265\n videoResource\n coverUrls {\n url\n __typename\n }\n timestamp\n expTag\n animatedCoverUrl\n distance\n videoRatio\n liked\n stereoType\n profileUserTopPhoto\n __typename\n}\n\nfragment feedContent on Feed {\n type\n author {\n id\n name\n headerUrl\n following\n headerUrls {\n url\n __typename\n }\n __typename\n }\n photo {\n ...photoContent\n __typename\n }\n canAddComment\n llsid\n status\n currentPcursor\n __typename\n}\n\nquery visionProfileLikePhotoList($pcursor: String, $page: String, $webPageArea: String) {\n visionProfileLikePhotoList(pcursor: $pcursor, page: $page, webPageArea: $webPageArea) {\n result\n llsid\n webPageArea\n feeds {\n ...feedContent\n __typename\n }\n hostName\n pcursor\n __typename\n }\n}\n"
}

发送请求获取的数据为json数据信息转换成字典数据&＃xff0c;从中提取出你想要的数据信息&＃xff0c;筛选出我们想要的视频标题&＃xff0c;以及视频的播放地址&＃xff0c;对视频地址发送请求就行&＃xff0c;想实现翻页效果需要注意&＃xff0c;我们下一页的页面数据是在他上个数据的接口里&＃xff0c;提取数据了时候一起提取出来

简易源码分享

import requestsdata &＃61; {"operationName": "visionProfileLikePhotoList","variables": {"pcursor": &＃39;&＃39;,"page": "profile"},"query": "fragment photoContent on PhotoEntity {\n id\n duration\n caption\n likeCount\n viewCount\n realLikeCount\n coverUrl\n photoUrl\n photoH265Url\n manifest\n manifestH265\n videoResource\n coverUrls {\n url\n __typename\n }\n timestamp\n expTag\n animatedCoverUrl\n distance\n videoRatio\n liked\n stereoType\n profileUserTopPhoto\n __typename\n}\n\nfragment feedContent on Feed {\n type\n author {\n id\n name\n headerUrl\n following\n headerUrls {\n url\n __typename\n }\n __typename\n }\n photo {\n ...photoContent\n __typename\n }\n canAddComment\n llsid\n status\n currentPcursor\n __typename\n}\n\nquery visionProfileLikePhotoList($pcursor: String, $page: String, $webPageArea: String) {\n visionProfileLikePhotoList(pcursor: $pcursor, page: $page, webPageArea: $webPageArea) {\n result\n llsid\n webPageArea\n feeds {\n ...feedContent\n __typename\n }\n hostName\n pcursor\n __typename\n }\n}\n"
}headers &＃61; {&＃39;accept&＃39;: &＃39;*/*&＃39;,&＃39;content-type&＃39;: &＃39;application/json&＃39;,&＃39;COOKIE&＃39;: &＃39;&＃39;,&＃39;Host&＃39;: &＃39;www.xxxxxxxxxx.com&＃39;,&＃39;Origin&＃39;: &＃39;https://www.xxxxxxxx.com&＃39;,&＃39;Referer&＃39;: &＃39;https://www.xxxxxxxx.com/profile/3x382umv98zjz79&＃39;,&＃39;User-Agent&＃39;: &＃39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36&＃39;,
}url &＃61; &＃39;https://www.xxxxxxxx.com/graphql&＃39;
res &＃61; requests.post(url, headers&＃61;headers, json&＃61;data)
for i in res.json()[&＃39;data&＃39;][&＃39;visionProfileLikePhotoList&＃39;][&＃39;feeds&＃39;]:# print(i)title &＃61; i[&＃39;photo&＃39;][&＃39;caption&＃39;]play_url &＃61; i[&＃39;photo&＃39;][&＃39;photoUrl&＃39;]res &＃61; requests.get(play_url)# print(res.content)title &＃61; title.replace(&＃39;#&＃39;, &＃39;&＃39;).replace(&＃39;&＃xff1f;&＃39;, &＃39;&＃39;).replace(&＃39;\n&＃39;, &＃39;&＃39;)with open(&＃39;某短视频/&＃39; &＃43; title &＃43; &＃39;.mp4&＃39;, &＃39;wb&＃39;)as f:f.write(res.content)print(&＃39;正在下载&＃xff1a;{}&＃39;.format(title))