如何用Java爬取笔趣阁(www.biquge5200.com)全部小说

目的
  现在很多年轻人都喜欢看网络小说,网络小说很容易成瘾,成瘾之后就得大把大把的向正版网站去订阅,你要不想订阅,只能去盗版网站去看,盗版网站上各种渣渣辉、古天乐的"是兄弟就来砍我"的页游广告,特别烦人。不想花钱去正版网站订阅,又不想看盗版网站的广告,怎么办?作为一个技术人,就要学会去爬虫,把盗版网站的数据爬过来,自己组织,给自己看(正版网站,需要订阅才能看,想要爬,得花钱————目前,我还没找到不花钱就可以爬的方法!)。(另外,盗版可耻,请大家关注正版,我这个只是为了学习技术练手需要才弄的)。

观察网页特点
  笔趣阁小说的全部小说栏目下,包含了笔趣阁小说的全部小说(实际上并不是全部,还有一些没有展示出来),点击对应小说的超链接,就可以进入到小说的详细目录,小说的详细目录下,包含了小说的章节信息,点击章节信息就可以获得小说的内容了。下面我们拿 龙血沸腾为例,来说:
1、全部小说页面的html

|  <!doctype html> |
|  | <head> |
|  | <title>小说大全_全部小说目录大全_笔趣阁</title> |
|  | <meta http-equiv="Content-Type" content="text/html; charset=gbk" /> |
|  | <meta name="keywords" content="小说大全" /> |
|  | <meta name="description" content="笔趣阁是广大书友最值得收藏的网络小说阅读网,笔趣阁网站收录了当前最火热的网络小说,笔趣阁5200免费提供高质量的小说最新章节,是广大网络小说爱好者必备的小说阅读网。" /> |
|  | <link rel="stylesheet" type="text/css" href="[http://r.biquge5200.com/images/biquge.css](http://r.biquge5200.com/images/biquge.css)"/> |
|  | <script type="text/javascript" src="[http://libs.baidu.com/jquery/1.4.2/jquery.min.js](http://libs.baidu.com/jquery/1.4.2/jquery.min.js)"></script> |
|  | <script type="text/javascript" src="[http://r.biquge5200.com/images/bqg.js](http://r.biquge5200.com/images/bqg.js)"></script> |
|  | </head> |
|  | <body> |
|  | <div id="wrapper"> |
|  | <script>login();</script> |
|  | <div class="header"> |
|  | <div class="header_logo"> |
|  | <a href="[http://www.biquge5200.com](http://www.biquge5200.com/)">笔趣阁</a> |
|  | </div> |
|  | <script>bqg_panel();</script> |
|  | </div> |
|  | <div class="nav"> |
|  | <ul> |
|  | <li><a href="[http://www.biquge5200.com/](http://www.biquge5200.com/)">首页</a></li> |
|  | <li><a rel="nofollow" href="[http://www.biquge5200.com/modules/article/bookcase.php](http://www.biquge5200.com/modules/article/bookcase.php)">我的书架</a></li> |
|  | <li><a href="[http://www.biquge5200.com/xuanhuanxiaoshuo/](http://www.biquge5200.com/xuanhuanxiaoshuo/)">玄幻小说</a></li> |
|  | <li><a href="[http://www.biquge5200.com/xiuzhenxiaoshuo/](http://www.biquge5200.com/xiuzhenxiaoshuo/)">修真小说</a></li> |
|  | <li><a href="[http://www.biquge5200.com/dushixiaoshuo/](http://www.biquge5200.com/dushixiaoshuo/)">都市小说</a></li> |
|  | <li><a href="[http://www.biquge5200.com/chuanyuexiaoshuo/](http://www.biquge5200.com/chuanyuexiaoshuo/)">穿越小说</a></li> |
|  | <li><a href="[http://www.biquge5200.com/wangyouxiaoshuo/](http://www.biquge5200.com/wangyouxiaoshuo/)">网游小说</a></li> |
|  | <li><a href="[http://www.biquge5200.com/kehuanxiaoshuo/](http://www.biquge5200.com/kehuanxiaoshuo/)">科幻小说</a></li> |
|  | <li><a href="[http://www.biquge5200.com/paihangbang/](http://www.biquge5200.com/paihangbang/)">排行榜单</a></li> |
|  | <li><a href="[http://www.biquge5200.com/xiaoshuodaquan/](http://www.biquge5200.com/xiaoshuodaquan/)">全部小说</a></li> |
|  | </ul> |
|  | </div> |
|  | <center><script>list1();</script></center> |
|  | <div class="MessageDiv"> |
|  | <b>提示:本站收录的全部小说均在此页, 推荐使用Ctrl+F 来查找小说。</b> |
|  | </div> |
|  | <div class="dahengfu"><script>list();</script></div> |
|  | <script>dingbu();</script> |
|  | <div id="main"> |
|  | <div class="novellist"> |
|  | <h2>奇幻、玄幻小说大全列表</h2> |
|  | <ul> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/14_14621/](http://www.biquge5200.com/14_14621/)">斗神狂飙</a><strong style="color:#FF0000">(完本)</strong></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/14_14620/](http://www.biquge5200.com/14_14620/)">龙血沸腾</a></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/14_14613/](http://www.biquge5200.com/14_14613/)">软妹异界游</a></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/14_14602/](http://www.biquge5200.com/14_14602/)">至尊仙魔</a></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/14_14601/](http://www.biquge5200.com/14_14601/)">丹武主宰</a></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/14_14598/](http://www.biquge5200.com/14_14598/)">斗仙机</a></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/14_14588/](http://www.biquge5200.com/14_14588/)">万妖至皇</a></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/14_14567/](http://www.biquge5200.com/14_14567/)">杀戮武皇</a></li> |
| <li><a href="[http://www.biquge5200.com/3_3316/](http://www.biquge5200.com/3_3316/)">最强超级英雄</a></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/3_3304/](http://www.biquge5200.com/3_3304/)">我的主神是团长</a></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/3_3293/](http://www.biquge5200.com/3_3293/)">阳世鬼差</a></li> |
|  |  |
|  | <li><a href="[http://www.biquge5200.com/3_3276/](http://www.biquge5200.com/3_3276/)">神的世界编辑器</a></li> |
|  | 
 |
|  | </ul></div> |
|  | <div class="clear"></div> |
|  | </div> |
|  | <div class="dahengfu"><script type="text/javascript">bottom();</script></div> |
|  |  |
|  | <div class="footer"> |
|  | <div class="footer_link"></div> |
|  | <div class="footer_cont"> |
|  | <script>footer();right();dl();</script> |
|  | </div> |
|  | </div> |
|  | </div> |
|  | </body> |
|  | </html> |

我们需要的小说详细页就在<li><a href="[http://www.biquge5200.com/14_14620/](http://www.biquge5200.com/14_14620/)">龙血沸腾</a></li> 里面,而
http://www.biquge5200.com/14_14620/就是小说的详细页面的URL了,使用正则表达式<li><a href="(?<detailurl>[\s\S]+?)">(?<bookname>[\s\S]+?)</a>就可以获得小说的标题及详细页链接了,但是会得到标题页的信息,后面要清理掉,另外还有红色强调的脏数据,后续也要清理掉。
好了,废话不说了,上代码!
1、导入需要的依赖的jar包,这个小程序是通过httpclient来实现的,所以要导入httpclient相关的jar包,maven依赖如下

<dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.3</version>
    </dependency>
    <dependency>
        <groupId>commons-codec</groupId>
        <artifactId>commons-codec</artifactId>
        <version>1.9</version>
    </dependency>
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
        <version>1.2</version>
    </dependency>
    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpcore</artifactId>
        <version>4.4.6</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>2.9.0</version>
    </dependency>
    <dependency>
        <groupId>net.sf.json-lib</groupId>
        <artifactId>json-lib</artifactId>
        <version>2.4</version>
    </dependency>
    <dependency>
        <groupId>commons-beanutils</groupId>
        <artifactId>commons-beanutils</artifactId>
        <version>1.8.0</version>
    </dependency>
    <dependency>
        <groupId>commons-collections</groupId>
        <artifactId>commons-collections</artifactId>
        <version>3.2.1</version>
    </dependency>
    <dependency>
        <groupId>commons-lang</groupId>
        <artifactId>commons-lang</artifactId>
        <version>2.5</version>
    </dependency>
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
        <version>1.1.1</version>
    </dependency>
    <dependency>
        <groupId>net.sf.ezmorph</groupId>
        <artifactId>ezmorph</artifactId>
        <version>1.0.6</version>
    </dependency>
    <dependency>
        <groupId>org.codehaus.groovy</groupId>
        <artifactId>groovy-all</artifactId>
        <version>1.7.5</version>
    </dependency>   
    <!-- https://mvnrepository.com/artifact/oro/oro -->
    <dependency>
        <groupId>oro</groupId>
        <artifactId>oro</artifactId>
        <version>2.0.8</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/xom/xom -->
    <dependency>
        <groupId>xom</groupId>
        <artifactId>xom</artifactId>
        <version>1.1</version>
    </dependency>
       <dependency>
      <!-- jsoup HTML parser library @ https://jsoup.org/ -->
      <groupId>org.jsoup</groupId>
      <artifactId>jsoup</artifactId>
      <version>1.11.2</version>
    </dependency>

为了过滤掉标题页的信息,我们使用Jsoup,直接获取<div class="novellist">的Element信息,代码如下:

    try {
            Document document = Jsoup.connect(Constants.BIQUGE5200_URL).get();  //public static final String BIQUGE5200_URL = "http://www.biquge5200.com/xiaoshuodaquan/";
            Elements element = document.getElementsByClass(Constants.BIQUGE5200_NOVELLIST_STR);  //public static final String BIQUGE5200_NOVELLIST_STR = "novellist";
            
        } catch (IOException e) {
            e.printStackTrace();
        }

获取到书籍信息列表的信息后,再通过正则表达式来获取书籍及对应的详情页url信息,代码如下:

        Pattern pattern = Pattern.compile(Constants.BIQUGE5200_ALL_GREP_RULE);
        try {
            Document document = Jsoup.connect(Constants.BIQUGE5200_URL).get();
            Elements element = document.getElementsByClass(Constants.BIQUGE5200_NOVELLIST_STR); //public static final String BIQUGE5200_ALL_GREP_RULE = "<li><a href=\"(?<detailurl>[\\s\\S]+?)\">(?<bookname>[\\s\\S]+?)</a>"; //笔趣阁5200全部小说页获取小说名称及url的正则表达式
            
            Matcher matcher = pattern.matcher(element.toString());
            while(matcher.find()){
                String novelname = matcher.group(2);
                String novelurl = matcher.group(1);
                
            }
            
        } catch (IOException e) {
            e.printStackTrace();
        }

得到章节URL之后,根据章节URL来获取章节内容,章节内容的页面很简单,只需要通过jsoup获取<div id="content">的内容即可,代码如下:

        Pattern pattern = Pattern.compile(Constants.BIQUGE5200_ALL_GREP_RULE);
        Pattern detail_pattern = Pattern.compile(Constants.BIQUGE5200_CHAPTER_GREP_RULE);
        try {
            Document document = Jsoup.connect(Constants.BIQUGE5200_URL).get();
            Elements element = document.getElementsByClass(Constants.BIQUGE5200_NOVELLIST_STR);
            Matcher matcher = pattern.matcher(element.toString());
            while(matcher.find()){
                String novelname = matcher.group(2);
                String novelurl = matcher.group(1);
                Document novel_detail_document = Jsoup.connect(novelurl).get();
                Element novel_detail_element = novel_detail_document.getElementById("list");
                Matcher detail = detail_pattern.matcher(novel_detail_element.toString());
                while(detail.find()){
                    String chaptername = detail.group(2);
                    String chapterurl = detail.group(1);
                    Document content_document = Jsoup.connect(chapterurl).get();
                    String content = content_document.getElementById("content").html().toString().replace("<br>", "");
                }
            }
            Thread.sleep(200l);
        } catch (IOException | InterruptedException e) {
            e.printStackTrace();
        }

最后将获取到的数据写入文件中,整体代码如下:
Constants.java

public class Constants {
    public static final String BIQUGE5200_URL = "http://www.biquge5200.com/xiaoshuodaquan/";   //笔趣阁5200 全部小说页的URL
    public static final String BIQUGE5200_ALL_GREP_RULE = "<li><a href=\"(?<detailurl>[\\s\\S]+?)\">(?<bookname>[\\s\\S]+?)</a>"; //笔趣阁5200全部小说页获取小说名称及url的正则表达式
    public static final String BIQUGE5200_CHAPTER_GREP_RULE = "<dd>\\s*<a href=\"(?<chapterurl>[\\s\\S]+?)\">(?<chaptername>[\\s\\S]+?)</a>\\s*</dd>"; //笔趣阁5200小说详细页获取
    public static final String BIQUGE5200_CONTENT_GREP_RULE = "";
    
    public static final String BIQUGE5200_NOVELLIST_STR = "novellist";
    
    public static final String BIQUGE5200_RESULT_DIR = "F:\\opt";
    
}

BiqugeSpider.java的代码:

public class BiqugeSpider {
    
    public static void main(String []args){
        Pattern pattern = Pattern.compile(Constants.BIQUGE5200_ALL_GREP_RULE);
        Pattern detail_pattern = Pattern.compile(Constants.BIQUGE5200_CHAPTER_GREP_RULE);
        try {
            Document document = Jsoup.connect(Constants.BIQUGE5200_URL).get();
            Elements element = document.getElementsByClass(Constants.BIQUGE5200_NOVELLIST_STR);
            Matcher matcher = pattern.matcher(element.toString());
            while(matcher.find()){
                String novelname = matcher.group(2);
                String novelurl = matcher.group(1);
                File file = new File(Constants.BIQUGE5200_RESULT_DIR + "\\" + novelname);
                if(file.isDirectory()||file.mkdir()){
                    Document novel_detail_document = Jsoup.connect(novelurl).get();
                    Element novel_detail_element = novel_detail_document.getElementById("list");
                    Matcher detail = detail_pattern.matcher(novel_detail_element.toString());
                    while(detail.find()){
                        String chaptername = detail.group(2);
                        String chapterurl = detail.group(1);
                        Document content_document = Jsoup.connect(chapterurl).get();
                        String content = content_document.getElementById("content").html().toString().replace("<br>", "");
                        String chapter_dir = Constants.BIQUGE5200_RESULT_DIR + "\\" + novelname + "\\" + chaptername + ".txt";
                        writeText(chapter_dir, content);
                    }
                }
                Thread.sleep(200l);
            }
        } catch (IOException | InterruptedException e) {
            e.printStackTrace();
        }
    }
    public static boolean writeText(String path, String text){
        File file = new File(path);
        boolean ret = true;
        try (FileOutputStream fos = new FileOutputStream(file);
             OutputStreamWriter osw = new OutputStreamWriter(fos, Charset.forName("UTF-8"));
             BufferedWriter bw = new BufferedWriter(osw, 4096)
            ){
            bw.append(text);
            bw.flush();
        } catch (IOException e) {
            ret = false;
        }
        return ret;
    }
    
}

遗留:
单线程跑,运行很慢,接下来要弄成多线程的,另外,目前拿到的小说,还不是全部的小说,还有一些是隐藏起来了,没有拿到,如果有需要的话,可以从起点中文网、创世中文网、纵横中文网、中文在线等正版网站中拿到书籍名称后,再到笔趣阁中来搜索,就可以拿到内容了。
搜索的URL组成如下:
http://www.biquge5200.com/modules/article/search.php?searchkey=书名 注意书名需要进行编码URLEncoder.encode("书名", "UTF-8");