16 Scrapy爬取二级目录

import scrapy
from ..items import SecondaryZufangItem

class ZufangSpider(scrapy.Spider):
    name = 'zufang'
    start_urls = ['http://bj.ganji.com/wblist/haidian/zufang/']
    #host_name = 'http://bj.ganji.com{}'

    def parse(self, response):
        zf = SecondaryZufangItem()
        next_link = response.xpath('//div[@class="pageBox"]/a[@class="next"]/@href').extract()[0]
        print(next_link,'*****'*20)
        yield scrapy.Request(next_link, callback = self.parse)

返回结果

SobersMacBookPro2016:secondary_zufang miraco$ scrapy crawl zufang
2018-12-20 14:58:24 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: secondary_zufang)
2018-12-20 14:58:24 [scrapy.utils.log] INFO: Versions: lxml 4.2.4.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.3 (v3.6.3:2c5fed86e0, Oct  3 2017, 00:32:08) - [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0i  14 Aug 2018), cryptography 2.3.1, Platform Darwin-17.7.0-x86_64-i386-64bit
2018-12-20 14:58:24 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'secondary_zufang', 'NEWSPIDER_MODULE': 'secondary_zufang.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['secondary_zufang.spiders']}
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-12-20 14:58:24 [scrapy.middleware] INFO: Enabled item pipelines:
['secondary_zufang.pipelines.SecondaryZufangPipeline']
2018-12-20 14:58:24 [scrapy.core.engine] INFO: Spider opened
2018-12-20 14:58:24 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-20 14:58:24 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-12-20 14:58:24 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/robots.txt> (referer: None)
2018-12-20 14:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/wblist/haidian/zufang/> (referer: None)
http://bj.ganji.com/haidian/zufang/pn2/ ****************************************************************************************************
2018-12-20 14:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn2/> (referer: http://bj.ganji.com/wblist/haidian/zufang/)
http://bj.ganji.com/haidian/zufang/pn3/ ****************************************************************************************************
2018-12-20 14:58:25 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn3/> (referer: http://bj.ganji.com/haidian/zufang/pn2/)
http://bj.ganji.com/haidian/zufang/pn4/ ****************************************************************************************************
2018-12-20 14:58:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn4/> (referer: http://bj.ganji.com/haidian/zufang/pn3/)
http://bj.ganji.com/haidian/zufang/pn5/ ****************************************************************************************************
2018-12-20 14:58:26 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn5/> (referer: http://bj.ganji.com/haidian/zufang/pn4/)
http://bj.ganji.com/haidian/zufang/pn6/ ****************************************************************************************************
2018-12-20 14:58:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn6/> (referer: http://bj.ganji.com/haidian/zufang/pn5/)
http://bj.ganji.com/haidian/zufang/pn7/ ****************************************************************************************************
2018-12-20 14:58:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn7/> (referer: http://bj.ganji.com/haidian/zufang/pn6/)
http://bj.ganji.com/haidian/zufang/pn8/ ****************************************************************************************************
2018-12-20 14:58:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn8/> (referer: http://bj.ganji.com/haidian/zufang/pn7/)
http://bj.ganji.com/haidian/zufang/pn9/ ****************************************************************************************************
2018-12-20 14:58:28 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn9/> (referer: http://bj.ganji.com/haidian/zufang/pn8/)
http://bj.ganji.com/haidian/zufang/pn10/ ****************************************************************************************************
2018-12-20 14:58:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn10/> (referer: http://bj.ganji.com/haidian/zufang/pn9/)
http://bj.ganji.com/haidian/zufang/pn11/ ****************************************************************************************************
2018-12-20 14:58:29 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn11/> (referer: http://bj.ganji.com/haidian/zufang/pn10/)
http://bj.ganji.com/haidian/zufang/pn12/ ****************************************************************************************************
2018-12-20 14:58:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn12/> (referer: http://bj.ganji.com/haidian/zufang/pn11/)
http://bj.ganji.com/haidian/zufang/pn13/ ****************************************************************************************************
2018-12-20 14:58:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn13/> (referer: http://bj.ganji.com/haidian/zufang/pn12/)
http://bj.ganji.com/haidian/zufang/pn14/ ****************************************************************************************************
2018-12-20 14:58:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn14/> (referer: http://bj.ganji.com/haidian/zufang/pn13/)
http://bj.ganji.com/haidian/zufang/pn15/ ****************************************************************************************************
2018-12-20 14:58:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn15/> (referer: http://bj.ganji.com/haidian/zufang/pn14/)
http://bj.ganji.com/haidian/zufang/pn16/ ****************************************************************************************************
2018-12-20 14:58:31 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn16/> (referer: http://bj.ganji.com/haidian/zufang/pn15/)
http://bj.ganji.com/haidian/zufang/pn17/ ****************************************************************************************************
2018-12-20 14:58:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn17/> (referer: http://bj.ganji.com/haidian/zufang/pn16/)
http://bj.ganji.com/haidian/zufang/pn18/ ****************************************************************************************************
2018-12-20 14:58:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn18/> (referer: http://bj.ganji.com/haidian/zufang/pn17/)
http://bj.ganji.com/haidian/zufang/pn19/ ****************************************************************************************************
2018-12-20 14:58:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn19/> (referer: http://bj.ganji.com/haidian/zufang/pn18/)
http://bj.ganji.com/haidian/zufang/pn20/ ****************************************************************************************************
2018-12-20 14:58:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn20/> (referer: http://bj.ganji.com/haidian/zufang/pn19/)
http://bj.ganji.com/haidian/zufang/pn21/ ****************************************************************************************************
2018-12-20 14:58:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn21/> (referer: http://bj.ganji.com/haidian/zufang/pn20/)
http://bj.ganji.com/haidian/zufang/pn22/ ****************************************************************************************************
2018-12-20 14:58:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn22/> (referer: http://bj.ganji.com/haidian/zufang/pn21/)
http://bj.ganji.com/haidian/zufang/pn23/ ****************************************************************************************************
2018-12-20 14:58:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn23/> (referer: http://bj.ganji.com/haidian/zufang/pn22/)
http://bj.ganji.com/haidian/zufang/pn24/ ****************************************************************************************************
2018-12-20 14:58:35 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn24/> (referer: http://bj.ganji.com/haidian/zufang/pn23/)
http://bj.ganji.com/haidian/zufang/pn25/ ****************************************************************************************************
2018-12-20 14:58:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://bj.ganji.com/haidian/zufang/pn25/> (referer: http://bj.ganji.com/haidian/zufang/pn24/)
http://bj.ganji.com/haidian/zufang/pn26/ ****************************************************************************************************
2018-12-20 14:58:36 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://bj.ganji.com/haidian/zufang/pn26/> (referer: http://bj.ganji.com/haidian/zufang/pn25/)
2018-12-20 14:58:36 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://bj.ganji.com/haidian/zufang/pn26/>: HTTP status code is not handled or not allowed
2018-12-20 14:58:36 [scrapy.core.engine] INFO: Closing spider (finished)

是吧，要将获取的url地址丢回调度器：

yield scrapy.Request(next_link[0], callback = self.parse)

这里不能写return，因为还要接着爬呢。

import scrapy
from ..items import SecondaryZufangItem

class ZufangSpider(scrapy.Spider):
    name = 'zufang'
    start_urls = ['http://bj.ganji.com/wblist/haidian/zufang/']
    #host_name = 'http://bj.ganji.com{}'

    def parse(self, response):
        zf = SecondaryZufangItem()
        title_list = response.xpath('.//div[@class="f-list-item ershoufang-list"]//dd[@class="dd-item title"]/a/text()').extract()
        money_list = response.xpath('.//span[@class="num"]/text()').extract()

        for i,j in zip(title_list,money_list):
            zf['title']=i
            zf['money']=j
            yield zf

        next_link = response.xpath('//div[@class="pageBox"]/a[@class="next"]/@href').extract()


        if len(next_link) > 0:
            print(next_link,'*****'*20)
            yield scrapy.Request(next_link[0], callback = self.parse)
        else:
            pass #自行添加，一般是处理验证页面啥的

返回的挺多呢

image.png

人面猴
序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
沈念sama阅读 159,716评论 4赞 364
死咒
序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
沈念sama阅读 67,558评论 1赞 294
救了他两次的神仙让他今天三更去死
文/潘晓璐我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
开封第一讲书人阅读 109,431评论 0赞 244
道士缉凶录：失踪的卖姜人
文/不坏的土叔我叫张陵，是天一观的道长。经常有香客问我，道长，这世上最难降的妖魔是什么？我笑而不...
开封第一讲书人阅读 44,127评论 0赞 209
港岛之恋（遗憾婚礼）
正文为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
茶点故事阅读 52,511评论 3赞 287
恶毒庶女顶嫁案：这布局不是一般人想出来的
文/花漫我一把揭开白布。她就那样静静地躺着，像睡着了一般。火红的嫁衣衬着肌肤如雪。梳的纹丝不乱的头发上，一...
开封第一讲书人阅读 40,692评论 1赞 222
城市分裂传说
那天，我揣着相机与录音，去河边找鬼。笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。我是一名探鬼主播，决...
沈念sama阅读 31,915评论 2赞 313
双鸳鸯连环套：你想象不到人心有多黑
文/苍兰香墨我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
开封第一讲书人阅读 30,664评论 0赞 202
万荣杀人案实录
序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
沈念sama阅读 34,412评论 1赞 246
护林员之死
正文独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋以下内容为张勋视角年9月15日...
茶点故事阅读 30,616评论 2赞 245
白月光启示录
正文我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
茶点故事阅读 32,105评论 1赞 260
活死人
序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
沈念sama阅读 28,424评论 2赞 254
日本核电站爆炸内幕
正文年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
茶点故事阅读 33,098评论 3赞 238
男人毒药：我在死后第九天来索命
文/蒙蒙一、第九天我趴在偏房一处隐蔽的房顶上张望。院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
开封第一讲书人阅读 26,096评论 0赞 8
一桩弑父案，背后竟有这般阴谋
文/苍兰香墨我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。一阵脚步声响...
开封第一讲书人阅读 26,869评论 0赞 197
情欲美人皮
我被黑心中介骗来泰国打工，没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。一个月前我还...
沈念sama阅读 35,748评论 2赞 276
代替公主和亲
正文我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
茶点故事阅读 35,641评论 2赞 271

16 Scrapy爬取二级目录

推荐阅读更多精彩内容