最近在爬区块链相关的快讯，上周巴比特改版后重写了爬虫，跑了一天就挂了。原来是网站使用了加速乐的服务，爬虫每次都返回521的状态码。
浏览器访问网站时：
第一次请求：返回521状态码和一段js代码。js会生成一段cookie并重新请求访问。
第二次请求：带着第一次得到的cookie去请求然后正确返回状态码200
而爬虫不能像浏览器一样执行js所以一直报错521
解决办法：
让爬虫模拟浏览器的行为：
将返回的js代码放在一个字符串中，然后利用execjs对这段代码进行解密，得到cookie放入下一次访问请求的头部中。
具体过程：
直接请求
将返回的这段js代码整理下：
<html>
<body>
<script language="javascript"> window.onload=setTimeout("ar(75)", 200);
function ar(YH) {
var qo, mo="", no="", oo = [0xc2,0x0c,0x22,0xa2,0x68,0x21,0xe8,0x3d,0x1e,0xbb,0x94,0x15,0x16,0x17,0x95,0x17,0x58,0x18,0xce,0xc6,0xc1,0xd6,0x16,0xb5,0x36,0xd6,0x96,0xd6,0xd0,0x2f,0x6f,0x50,0xd0,0x90,0x47,0x18,0xcd,0xa3,0x39,0x57,0x37,0x77,0x89,0x49,0x47,0x9d,0xdd,0x14,0x6a,0xab,0x8b,0x81,0x3f,0x15,0x4c,0xc2,0x49,0x68,0x08,0x1f,0x36,0xb6,0xec,0xaa,0x63,0x39,0x57,0xd7,0x6d,0x26,0x08,0x9d,0x1e,0x74,0x8b,0x44,0x84,0xb1,0x8f,0xe5,0x1d,0xd5,0xec,0x8c,0xa0,0xe0,0x18,0xd7,0x0f,0x46,0xe5,0x23,0x00,0xb6,0x37,0xb7,0x70,0xa6,0x4e,0x04,0x7a,0x18,0x0e,0xc3,0x79,0x4a,0x68,0xbe,0x74,0xeb,0x04,0xc3,0x67,0x86,0xa4,0xe5,0x44,0x04,0x82,0xcb,0x82,0x47,0x48,0x21,0xb9,0xd1,0xfa,0x51,0x6f,0x28,0x64,0x22,0x22,0xc0,0x71,0xaf,0xc6,0xde,0xf4,0x0c,0xd4,0x2c,0xe1,0xff,0x57,0xad,0x63,0x8c,0xa4,0xa8,0x65,0x07,0x7e,0x96,0xa7,0x47,0x48,0x01,0x41,0x82,0x63,0x33,0xe9,0xc2,0xd9,0x3a,0xdf,0x60,0x73,0x4c,0xcc,0xcd,0x8e,0x06,0x1e,0x1b,0x39,0x79,0x1f,0x40,0xf6,0xef,0xa3,0x9b,0x13,0x2b,0x29,0x6a,0x4b,0x6b,0x0b,0x0c,0x0a,0xe2,0x82,0x83,0x27,0xa7,0x65,0x26,0xe5,0xc6,0x64,0xef,0xc8,0x61,0x62,0xe2,0x23,0xc8,0xd0,0x0a,0x0b,0xeb,0xa2,0x42,0x43,0xee,0x6f,0x2d,0xed,0xad,0x8e,0x2c,0xfc,0xd5,0x97,0xf1,0xf0,0x3b];
qo = "qo=228; do{oo[qo]=(-oo[qo])&0xff; oo[qo]=(((oo[qo]>>3)|((oo[qo]<<5)&0xff))-251)&0xff;} while(--qo>=2);";
eval(qo);
qo = 227;
do {
oo[qo] = (oo[qo] - oo[qo - 1]) & 0xff;
} while (-- qo >= 3 );
qo = 1;
for (;;) {
if (qo > 227) break;
oo[qo] = ((((((oo[qo] + 28) & 0xff) + 148) & 0xff) << 6) & 0xff) | (((((oo[qo] + 28) & 0xff) + 148) & 0xff) >> 2);
qo++;
}
po = "";
for (qo = 1; qo < oo.length - 1; qo++) if (qo % 7) po += String.fromCharCode(oo[qo] ^ YH);
eval("qo=eval;qo(po);");
}
</script>
</body>
</html>
然后存为html文件用Chrome打开调试，在eval处打上断点。可以看到变量po的值："document.cookie='_ydclearance=5640fae72a12f756938d88c1-60c4-4c28-a629-8da9e99d65cc-1534755025; expires=Mon, 20-Aug-18 08:50:25 GMT; domain=.8btc.com; path=/'; window.document.location=document.URL"
而字符串po的前半段的意思是给浏览器添加Cooklie,后半段window.document.location=document.URL是刷新当前页面。
所以我们的关键点是要获得cookie。python中可以用execjs执行js:
import requests
import re
import execjs
headers = {
'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/65.0.3325.181 Chrome/65.0.3325.181 Safari/537.36',
}
def get_html(url):
first_html = requests.get(url=url,headers=headers).content.decode('utf-8')
return first_html
def executejs(first_html):
# 提取其中的JS加密函数
js_string = ''.join(re.findall(r'(function .*?)</script>', first_html))
# 提取其中执行JS函数的参数
js_arg = ''.join(re.findall(r'setTimeout\(\"\D+\((\d+)\)\"', first_html))
js_name = re.findall(r'function (\w+)',js_string)[0]
# 修改JS函数，使其返回Cookie内容
js_string = js_string.replace('eval("qo=eval;qo(po);")', 'return po')
func = execjs.compile(js_string)
return func.call(js_name,js_arg)
def parse_cookie(string):
string = string.replace("document.cookie='", "")
clearance = string.split(';')[0]
return {clearance.split('=')[0]: clearance.split('=')[1]}
def return_cookie(url):
first_html = get_html(url)
# 执行JS获取Cookie
cookie_str = executejs(first_html)
# 将Cookie转换为字典格式
cookie = parse_cookie(cookie_str)
print('cookies = ',cookie)
return cookie
return_cookie(url='https://www.8btc.com/flash')
#结果：
cookies = {'_ydclearance': '8c83e7fe9d6bd359e1eedc40-b55a-4ab5-98e2-22eb9b2ea9a7-1534917111'}
[Finished in 2.0s]
