403 Forbidden
The server understood the request but refuses to authorize it.
Troubleshooting
在setting.py文件中增加USER_AGENT配置：
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
Others
查看 Chrome 浏览器的 User-Agent
shift+ctrl+c, 打开chrome自带的调试工具(这和 F12 有点差别);选择 network 标签, 刷新网页(在打开调试工具的情况下刷新);
刷新后在左边找到该网页url,点击后右边选择 headers ,找到 Request-headers 下的 User-Agent;
在Scrapy shell 中设置 User-Agent
scrapy shell 'xxx_url' -s USER_AGENT='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
