jsoup其实只是一种网页分析器，帮助java程序员进行网页元素分析，其代替了用正则表达式去匹配信息的方式，效率更高也跟容易编写。
分析需求
从51job上爬取职位信息，然后记录下来
步骤
爬虫的步骤无非就是下载网页，分析网络，获取信息，继续爬新的网页。来看下jsoup的代码。
下载网页
jsoup对下载网页的过程进行了封装，设置完一些http参数之后执行，最终返回一个Document对象，这个对象其实就是一个DOM树，他继承Element。
Document document = Jsoup.connect(url)//连接url
.userAgent("ie7：mozilla/4.0 (compatible; msie 7.0b; windows nt 6.0)")//模拟浏览器访问
.timeout(3000)//设置超时
.cookie("guide", "1")//一个坑
.followRedirects(false)//是否跳转
.execute().parse();//执行
在做这个网页时，遇到了一个坑。在jsoup的过程中发现一直抓不到正确的页面，后来发现抓到的是他的引导页面。就抓了一个浏览器的http包，查看了一下cookie。猜测了一下应该是guide这个参数了，于是.cookie("guide", "1”)，再次获取页面，获取正常！
然后利用jsoup抓取信息
没有正则表达式感觉好清晰~由于Document中存了各种标签的信息，直接获取标签就可以了。
job.setJobID(element.getElementsByClass("checkbox").first().attr("value"));
Element t1 = element.getElementsByClass("t1").first();
job.setJobTitle(t1.getElementsByTag("a").attr("title"));
job.setJobDetailUrl(t1.getElementsByTag("a").attr("href"));
Element t2 = element.getElementsByClass("t2").first();
job.setCompanyName(t2.getElementsByTag("a").attr("title"));
job.setLocation(element.getElementsByClass("t3").text());
job.setSalary(element.getElementsByClass("t4").text());
job.setDate(element.getElementsByClass("t5").text());
jobs.add(job);```
jsoup支持CSS抓取方式，这里没有体现，其大概用法如下。
// 使用select方法选择元素，参数是CSS Selector表达式
Elements links = doc.select("a[href]");
print("\nLinks: (%d)", links.size());
for (Element link : links) {
//使用abs:前缀取绝对url地址
print(" * a: <%s> (%s)", link.attr("abs:href"), trim(link.text(), 35));
}
源码在git上：https://github.com/bingochaos/Spider51job
#jsoup源码阅读
###如何解析成DOM树
其实解析html就是一个词法分析的过程。阅读标签头然后存储，找到标签尾结束当前标签。看到jsoup的实现的时候我还是惊了个呆。
enum TokeniserState {
Data {
// in data state, gather characters until a character reference or tag is found
void read(Tokeniser t, CharacterReader r) {
switch (r.current()) {
case '&':
t.advanceTransition(CharacterReferenceInData);
break;
case '<':
t.advanceTransition(TagOpen);
break;
case nullChar:
t.error(this); // NOT replacement character (oddly?)
t.emit(r.consume());
break;
case eof:
t.emit(new Token.EOF());
break;
default:
String data = r.consumeData();
t.emit(data);
break;
}
}
},
CharacterReferenceInData {
// from & in data
void read(Tokeniser t, CharacterReader r) {
char[] c = t.consumeCharacterReference(null, false);
if (c == null)
t.emit('&');
else
t.emit(c);
t.transition(Data);
}
},
大部分略。。。
jsoup利用枚举来表示状态，每个状态需要执行的代码放在了枚举里面,并用下面这种方式完成了状态机的转化。
while (!isEmitPending)
state.read(this, reader);
这里盗图一张来说明这个词法分析的过程
#####缺少标签会发生什么事？
答案是很多会当做处理被处理掉，代码如下
case EndTag:
if (StringUtil.in(name,"div","dl", "fieldset", "figcaption", "figure", "footer", "header", "pre", "section", "summary", "ul")) {
if (!tb.inScope(name)) {
tb.error(this);
return false;
}
}
#####最后附上jsoup对tag的分类
// internal static initialisers:
// prepped from http://www.w3.org/TR/REC-html40/sgml/dtd.html and other sources
private static final String[] blockTags = {
"html", "head", "body", "frameset", "script", "noscript", "style", "meta", "link", "title", "frame",
"noframes", "section", "nav", "aside", "hgroup", "header", "footer", "p", "h1", "h2", "h3", "h4", "h5", "h6",
"ul", "ol", "pre", "div", "blockquote", "hr", "address", "figure", "figcaption", "form", "fieldset", "ins",
"del", "s", "dl", "dt", "dd", "li", "table", "caption", "thead", "tfoot", "tbody", "colgroup", "col", "tr", "th",
"td", "video", "audio", "canvas", "details", "menu", "plaintext", "template", "article", "main",
"svg", "math"
};
private static final String[] inlineTags = {
"object", "base", "font", "tt", "i", "b", "u", "big", "small", "em", "strong", "dfn", "code", "samp", "kbd",
"var", "cite", "abbr", "time", "acronym", "mark", "ruby", "rt", "rp", "a", "img", "br", "wbr", "map", "q",
"sub", "sup", "bdo", "iframe", "embed", "span", "input", "select", "textarea", "label", "button", "optgroup",
"option", "legend", "datalist", "keygen", "output", "progress", "meter", "area", "param", "source", "track",
"summary", "command", "device", "area", "basefont", "bgsound", "menuitem", "param", "source", "track",
"data", "bdi"
};
private static final String[] emptyTags = {
"meta", "link", "base", "frame", "img", "br", "wbr", "embed", "hr", "input", "keygen", "col", "command",
"device", "area", "basefont", "bgsound", "menuitem", "param", "source", "track"
};
private static final String[] formatAsInlineTags = {
"title", "a", "p", "h1", "h2", "h3", "h4", "h5", "h6", "pre", "address", "li", "th", "td", "script", "style",
"ins", "del", "s"
};
private static final String[] preserveWhitespaceTags = {
"pre", "plaintext", "title", "textarea"
// script is not here as it is a data node, which always preserve whitespace
};
// todo: I think we just need submit tags, and can scrub listed
private static final String[] formListedTags = {
"button", "fieldset", "input", "keygen", "object", "output", "select", "textarea"
};
private static final String[] formSubmitTags = {
"input", "keygen", "object", "select", "textarea"
};
