利用Python进行简单的一些图片网站爬虫。

我们分为三部分来完成这个爬虫
获取页数的url - 解析页面的HTLM - 下载图片

PAGE的url是这样的:www.91doutu.com/category/qq表情包/page/1
我们可以用for循环来遍历出我们需要爬虫的页数。

BASE_PAGE_URL = 'http://www.91doutu.com/category/qq%E8%A1%A8%E6%83%85%E5%8C%85/page/'
for i in range(0,11):
    print BASE_PAGE_URL + str(i)

这样就获取到了我们需要的page_url了。

PAGE_URL

接下来我们来完成第二步

解析页面的HTML源码 获取我们需要的部分。

#encoding
import requests
from bs4 import BeautifulSoup

response = requests.get('http://www.91doutu.com/category/qq%E8%A1%A8%E6%83%85%E5%8C%85')
content = response.content
soup = BeautifulSoup(content,'lxml')
img_list = soup.find_all('img',attrs={'class':'thumb'})
for img in img_list:
    print img['data-src']

这样就获取到了我们需要的图片url了。

Img_List

第三步-下载

只需要用到一个函数就轻轻松松搞定。

首先分割url 取list最后一个元素来当做我们的文件名,然后再下载到images目录下。

#encoding
import requests
from bs4 import BeautifulSoup
import os
import urllib

def download_image(url):
    split_list = url.split('/')
    filename = split_list.pop()
    path = os.path.join('images',filename)
    urllib.urlretrieve(url,filename=path)

response = requests.get('http://www.91doutu.com/category/qq%E8%A1%A8%E6%83%85%E5%8C%85')
content = response.content
soup = BeautifulSoup(content,'lxml')
img_list = soup.find_all('img',attrs={'class':'thumb'})
for img in img_list:
    url = img['data-src']
    download_image(url)

download_img

完整的Code：

#encoding
#_PlugName_ = Spider_Img
#__Author__ = Search__
# @Time    : 2017/8/29
#__Refer___ = http://www.jianshu.com/u/d743d12d1d77

import requests
from bs4 import BeautifulSoup
import os
import urllib

BASE_PAGE_URL = 'http://www.91doutu.com/category/qq%E8%A1%A8%E6%83%85%E5%8C%85/page/'
PAGE_URL_LIST = []
for x in range(7,10):
    url = BASE_PAGE_URL + str(x)
    PAGE_URL_LIST.append(url)

def download_image(url):
    split_list = url.split('/')
    filename = split_list.pop()
    path = os.path.join('images',filename)
    urllib.urlretrieve(url,filename=path)

def get_page(page_url):
    response = requests.get(page_url)
    content = response.content
    soup = BeautifulSoup(content,'lxml')
    img_list = soup.find_all('img',attrs={'class':'thumb'})
    for img in img_list:
        url = img['data-src']
        download_image(url)

def main():
    for page_url in PAGE_URL_LIST:
        get_page(page_url)

if __name__ == "__main__":
    main()