作者:鲁野山人2602886435 | 来源:互联网 | 2022-12-09 08:54
我想在Go中下载Fantasy Football Data进行分析,但是当我尝试从这个api页面下载时,我得到了一个空的响应,即使该代码适用于其他网站,例如这个api页面
最小的再现,输出一个空数组.
package main
import (
"fmt"
"io/ioutil"
"net/http"
"time"
)
const AllPlayerData = "https://fantasy.premierleague.com/drf/bootstrap-static"
func main() {
downloadAllData()
}
func downloadAllData() {
client := &http.Client{
Timeout: 20 * time.Second,
}
response, err := client.Get(AllPlayerData)
if err != nil {
fmt.Println("Unable to download player data.")
return
}
body, err := ioutil.ReadAll(response.Body)
if err != nil {
fmt.Println("Failed to read response")
return
}
defer response.Body.Close()
fmt.Println(body)
}
相同的网页在Python中下载得很好:
import requests
url = "https://fantasy.premierleague.com/drf/bootstrap-static"
r = requests.get(url)
print(r.content)
我不认为它与例如Ajax调用有关,因为在Chrome中查看网络请求并不会超出页面加载本身
1> dave..:
他们正在对用户代理进行某种验证,以下代码有效:
package main
import (
"fmt"
"io/ioutil"
"net/http"
"time"
)
const AllPlayerData = "https://fantasy.premierleague.com/drf/bootstrap-static"
func main() {
downloadAllData()
}
func downloadAllData() {
client := &http.Client{
Timeout: 20 * time.Second,
}
request, err := http.NewRequest(http.MethodGet, AllPlayerData, nil)
if err != nil {
fmt.Println("Unable to create request.")
return
}
request.Header.Add("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36")
response, err := client.Do(request)
if err != nil {
fmt.Println("Unable to download player data.")
return
}
body, err := ioutil.ReadAll(response.Body)
if err != nil {
fmt.Println("Failed to read response")
return
}
defer response.Body.Close()
fmt.Println(string(body))
}
Go实际上默认也有一个用户代理,`const defaultUserAgent ="Go-http-client/1.1"`,我想PostMan要么有一个,要么使用你的浏览器.很难知道*为什么*Go的默认设置被阻止,但这只是抓取的常见问题 - 通常你想尽可能地模拟真实的浏览器