利用jsoup爬虫

jsoup其实只是一种网页分析器,帮助java程序员进行网页元素分析,其代替了用正则表达式去匹配信息的方式,效率更高也跟容易编写。

分析需求

从51job上爬取职位信息,然后记录下来

步骤

爬虫的步骤无非就是下载网页,分析网络,获取信息,继续爬新的网页。来看下jsoup的代码。

下载网页

jsoup对下载网页的过程进行了封装,设置完一些http参数之后执行,最终返回一个Document对象,这个对象其实就是一个DOM树,他继承Element。

        Document document = Jsoup.connect(url)//连接url
                .userAgent("ie7:mozilla/4.0 (compatible; msie 7.0b; windows nt 6.0)")//模拟浏览器访问
                .timeout(3000)//设置超时
                .cookie("guide", "1")//一个坑
                .followRedirects(false)//是否跳转
                .execute().parse();//执行

在做这个网页时,遇到了一个坑。在jsoup的过程中发现一直抓不到正确的页面,后来发现抓到的是他的引导页面。就抓了一个浏览器的http包,查看了一下cookie。猜测了一下应该是guide这个参数了,于是.cookie("guide", "1”),再次获取页面,获取正常!


抓到的包

然后利用jsoup抓取信息

没有正则表达式感觉好清晰~由于Document中存了各种标签的信息,直接获取标签就可以了。

           job.setJobID(element.getElementsByClass("checkbox").first().attr("value"));
            Element t1 = element.getElementsByClass("t1").first();
            job.setJobTitle(t1.getElementsByTag("a").attr("title"));
            job.setJobDetailUrl(t1.getElementsByTag("a").attr("href"));
            Element t2 = element.getElementsByClass("t2").first();
            job.setCompanyName(t2.getElementsByTag("a").attr("title"));
            job.setLocation(element.getElementsByClass("t3").text());
            job.setSalary(element.getElementsByClass("t4").text());
            job.setDate(element.getElementsByClass("t5").text());
            jobs.add(job);```
jsoup支持CSS抓取方式,这里没有体现,其大概用法如下。

// 使用select方法选择元素,参数是CSS Selector表达式
Elements links = doc.select("a[href]");

    print("\nLinks: (%d)", links.size());
    for (Element link : links) {
        //使用abs:前缀取绝对url地址
        print(" * a: <%s>  (%s)", link.attr("abs:href"), trim(link.text(), 35));
    }
源码在git上:https://github.com/bingochaos/Spider51job

#jsoup源码阅读
###如何解析成DOM树
其实解析html就是一个词法分析的过程。阅读标签头然后存储,找到标签尾结束当前标签。看到jsoup的实现的时候我还是惊了个呆。

enum TokeniserState {
Data {
// in data state, gather characters until a character reference or tag is found
void read(Tokeniser t, CharacterReader r) {
switch (r.current()) {
case '&':
t.advanceTransition(CharacterReferenceInData);
break;
case '<':
t.advanceTransition(TagOpen);
break;
case nullChar:
t.error(this); // NOT replacement character (oddly?)
t.emit(r.consume());
break;
case eof:
t.emit(new Token.EOF());
break;
default:
String data = r.consumeData();
t.emit(data);
break;
}
}
},
CharacterReferenceInData {
// from & in data
void read(Tokeniser t, CharacterReader r) {
char[] c = t.consumeCharacterReference(null, false);
if (c == null)
t.emit('&');
else
t.emit(c);
t.transition(Data);
}
},
大部分略。。。

jsoup利用枚举来表示状态,每个状态需要执行的代码放在了枚举里面,并用下面这种方式完成了状态机的转化。

while (!isEmitPending)
state.read(this, reader);

这里盗图一张来说明这个词法分析的过程

![jsoup词法分析过程](http://upload-images.jianshu.io/upload_images/1677321-2a3886f9fd8c765d.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

#####缺少标签会发生什么事?
答案是很多会当做处理被处理掉,代码如下
case EndTag:
    if (StringUtil.in(name,"div","dl", "fieldset", "figcaption", "figure", "footer", "header", "pre", "section", "summary", "ul")) {                
        if (!tb.inScope(name)) {
        tb.error(this);
        return false;
        } 
    }  
#####最后附上jsoup对tag的分类
// internal static initialisers:
// prepped from http://www.w3.org/TR/REC-html40/sgml/dtd.html and other sources
private static final String[] blockTags = {
        "html", "head", "body", "frameset", "script", "noscript", "style", "meta", "link", "title", "frame",
        "noframes", "section", "nav", "aside", "hgroup", "header", "footer", "p", "h1", "h2", "h3", "h4", "h5", "h6",
        "ul", "ol", "pre", "div", "blockquote", "hr", "address", "figure", "figcaption", "form", "fieldset", "ins",
        "del", "s", "dl", "dt", "dd", "li", "table", "caption", "thead", "tfoot", "tbody", "colgroup", "col", "tr", "th",
        "td", "video", "audio", "canvas", "details", "menu", "plaintext", "template", "article", "main",
        "svg", "math"
};
private static final String[] inlineTags = {
        "object", "base", "font", "tt", "i", "b", "u", "big", "small", "em", "strong", "dfn", "code", "samp", "kbd",
        "var", "cite", "abbr", "time", "acronym", "mark", "ruby", "rt", "rp", "a", "img", "br", "wbr", "map", "q",
        "sub", "sup", "bdo", "iframe", "embed", "span", "input", "select", "textarea", "label", "button", "optgroup",
        "option", "legend", "datalist", "keygen", "output", "progress", "meter", "area", "param", "source", "track",
        "summary", "command", "device", "area", "basefont", "bgsound", "menuitem", "param", "source", "track",
        "data", "bdi"
};
private static final String[] emptyTags = {
        "meta", "link", "base", "frame", "img", "br", "wbr", "embed", "hr", "input", "keygen", "col", "command",
        "device", "area", "basefont", "bgsound", "menuitem", "param", "source", "track"
};
private static final String[] formatAsInlineTags = {
        "title", "a", "p", "h1", "h2", "h3", "h4", "h5", "h6", "pre", "address", "li", "th", "td", "script", "style",
        "ins", "del", "s"
};
private static final String[] preserveWhitespaceTags = {
        "pre", "plaintext", "title", "textarea"
        // script is not here as it is a data node, which always preserve whitespace
};
// todo: I think we just need submit tags, and can scrub listed
private static final String[] formListedTags = {
        "button", "fieldset", "input", "keygen", "object", "output", "select", "textarea"
};
private static final String[] formSubmitTags = {
        "input", "keygen", "object", "select", "textarea"
};

推荐阅读更多精彩内容

  • Android 自定义View的各种姿势1 Activity的显示之ViewRootImpl详解 Activity...
    passiontim阅读 160,229评论 24 690
  • HTTP cookie(也称为web cookie,网络cookie,浏览器cookie或者简称cookie)是网...
    留七七阅读 14,068评论 2 71
  • Spring Cloud为开发人员提供了快速构建分布式系统中一些常见模式的工具(例如配置管理,服务发现,断路器,智...
    卡卡罗2017阅读 123,403评论 18 134
  • HTML标签解释大全 一、HTML标记 标签:!DOCTYPE 说明:指定了 HTML 文档遵循的文档类型定义(D...
    米塔塔阅读 2,568评论 1 41
  • 最忆往昔时, 言与心中月。 君之在我心, 犹似月无阙。 乞君常欢颜, 我心乐常奏。 奈何自性劣, 夕夕离雅娴。 反...
    任平梵阅读 97评论 0 0
  • 近来,随着思绪的飘飞,我也开始认定:或许真的,你并不是我命中注定的那个人,但即便如此,我也无法去爱上一个除你以外的...
    奇奇怪怪小技巧阅读 89评论 9 4
  • 文/wifi过 客 图/网 络 在领读者~新疆大厦里,读完了这本全世界非常畅...
    wifi过客阅读 428评论 0 1