尝试Selenium分布式爬虫

探究

想法是,手头上有2台服务器。如果有爬抓数据任务,这两台机器同时工作并且最大程度上利用网络和多核CPU资源我又通过multiprocess开启多进程,提高速度。我在之前抓京东数据时做了一份统计
1进程 147秒
2进程 67秒
3进程 58秒
4进程 56秒
我本身的机器是4核,用的是公司的百兆光纤网。当然分配给每一个IP肯定做了速度限制。
但是从上面的数据可以看出,多核还是非常有用的。假设你的任务queue,那么多核抓取也非常合适的代码类似于这样,可以参考我的做法。

p = Pool(3)
for item in items:
        p.apply_async(crwal, args=(item,))
 print('Waiting for all subprocesses done...')
 p.close()
 p.join()

上面的也不算废话,想象一下如果使用2台机器去跑,那么抓取时间理论上减半。3台、4台....N台 ,抓取时间就是 time/N 那么我们实现分布式部署一个可拓展,稳定,容错的可靠系统还是非常有必要的。
那么分布式到底如何去做呢?我在网上查阅了一些资料,查到Python有个分布式模型Celery,看下它的架构图。

Celery架构图

User 用户即使用者
broker 消息中间人Celery 需要一个发送和接收消息的解决方案,其通常以独立服务形式出现, 仔细观察实习AMQP,这是高级消息队列协议的缩写。实现这个协议的MQ非常多,官网主推RabbitMQ和Redis ,而前者更是推荐,因为后者容易受到突然断电会丢失数据。请看中文官网解释
workers 工作机
task result store 结果存储。这样的好处是完全分离。互不依赖。
因此Celery+RabbitMQ的结合架构是这样子的(图片来源于网络)
image.png

RabbitMQ安装

系统环境:CentOS7.4

yum -y install erlang

下载rpm 若链接不上,建议直接复制地址到迅雷下载,会很快。然后再rz导入到系统里即可。

wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-3.6.6-1.el7.noarch.rpm

安装

yum install rabbitmq-server-3.6.6-1.el7.noarch.rpm 

启动

sudo systemctl start rabbitmq-server

报错类似于

-- Subject: Unit rabbitmq-server.service has begun start-up
-- Defined-By: systemd
   Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
▽- 
-- Unit rabbitmq-server.service has begun starting up.
Jul 10 14:21:20 l22-30-91 rabbitmq-server[6275]: ERROR: epmd error for host l22-30-91: address (cannot connect to host/port)
Jul 10 14:21:20 l22-30-91 systemd[1]: rabbitmq-server.service: main process exited, code=exited, status=1/FAILURE
Jul 10 14:21:20 l22-30-91 systemd[1]: Cannot find unit for notify message of PID 6556.
Jul 10 14:21:21 l22-30-91 rabbitmqctl[6447]: Stopping and halting node 'rabbit@l22-30-91' ...
Jul 10 14:21:24 l22-30-91 rabbitmqctl[6447]: Error: unable to connect to node 'rabbit@l22-30-91': nodedown
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: DIAGNOSTICS
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: ===========
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: attempted to contact: ['rabbit@l22-30-91']
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: rabbit@l22-30-91:
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: * unable to connect to epmd (port 4369) on l22-30-91: address (cannot connect to host/port)
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: current node details:
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: - node name: 'rabbitmq-cli-09@l22-30-91'
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: - home dir: .
Jul 10 14:21:27 l22-30-91 rabbitmqctl[6447]: - cookie hash: xTHTDK3u5qC0QMJxWs38NQ==
Jul 10 14:21:27 l22-30-91 systemd[1]: Failed to start RabbitMQ broker.
-- Subject: Unit rabbitmq-server.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit rabbitmq-server.service has failed.
-- 
-- The result is failed.
Jul 10 14:21:27 l22-30-91 systemd[1]: Unit rabbitmq-server.service entered failed state.
Jul 10 14:21:27 l22-30-91 systemd[1]: rabbitmq-server.service failed.
Jul 10 14:21:27 l22-30-91 polkitd[477]: Unregistered Authentication Agent for unix-process:6259:33349831 (system bus name :1.12504, object path /org/freedesktop/Polic

这段异常有一个地方很重要。
Jul 10 14:21:20 l22-30-91 rabbitmq-server[6275]: ERROR: epmd error for host l22-30-91: address (cannot connect to host/port)
记住 l22-30-91 这是你的hostname

vim /etc/hosts
在里面新加一行
127.0.0.1 your-hostname

设置开机启动,并启动即可,查看状态

sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server
sudo systemctl stop rabbitmq-server

打印出

 # sudo systemctl status rabbitmq-server     
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-07-10 14:22:51 CST; 39min ago
  Process: 6447 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
 Main PID: 6744 (beam.smp)
   Status: "Initialized"
   Memory: 45.9M
   CGroup: /system.slice/rabbitmq-server.service
           ├─6744 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -A 128 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -- -root /usr/lib64/erlang -progname e...
           ├─6889 /usr/lib64/erlang/erts-5.10.4/bin/epmd -daemon
           ├─7097 inet_gethost 4
           └─7098 inet_gethost 4

Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: RabbitMQ 3.6.6. Copyright (C) 2007-2016 Pivotal Software, Inc.
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ##  ##
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ##########  Logs: /var/log/rabbitmq/rabbit@l22-30-91.log
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ######  ##        /var/log/rabbitmq/rabbit@l22-30-91-sasl.log
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: ##########
Jul 10 14:22:50 l22-30-91 rabbitmq-server[6744]: Starting broker...
Jul 10 14:22:51 l22-30-91 rabbitmq-server[6744]: systemd unit for activation check: "rabbitmq-server.service"
Jul 10 14:22:51 l22-30-91 systemd[1]: Started RabbitMQ broker.
Jul 10 14:22:51 l22-30-91 rabbitmq-server[6744]: completed with 0 plugins.

启用rabbitmq-management插件:

sudo rabbitmq-plugins enable rabbitmq_management

重启生效

sudo systemctl restart rabbitmq-server

推荐阅读更多精彩内容