摘要
本节讲解
快照文件意义
SnapShot接口
FileSnap源码
快照文件可视化
意义
Zookeeper的数据在内存中是以DataTree为数据结构存储的
而快照就是每间隔一段时间Zookeeper就会把整个DataTree的数据序列化然后把它存储在磁盘中
快照文件是指定时间间隔对数据的备份，所以快照文件中数据通常都不是最新的
多久抓一个快照这也是可以配置的snapCount配置项用于配置处理几个事务请求后生成一个快照文件；
接口SnapShot
是持久化中快照snapshot的接口
public interface SnapShot {
/**
* deserialize a data tree from the last valid snapshot and
* return the last zxid that was deserialized
* @param dt the datatree to be deserialized into
* @param sessions the sessions to be deserialized into
* @return the last zxid that was deserialized from the snapshot
* @throws IOException
*/
// 反序列化
long deserialize(DataTree dt, Map<Long, Integer> sessions)
throws IOException;
/**
* persist the datatree and the sessions into a persistence storage
* @param dt the datatree to be serialized
* @param sessions
* @throws IOException
*/
//序列化
void serialize(DataTree dt, Map<Long, Integer> sessions,
File name)
throws IOException;
/**
* find the most recent snapshot file
* @return the most recent snapshot file
* @throws IOException
*/
//找到最近的快照文件
File findMostRecentSnapshot() throws IOException;
/**
* free resources from this snapshot immediately
* @throws IOException
*/
//释放资源
void close() throws IOException;
}
函数分别是
序列化，反序列化,找到最近的快照文件以及释放资源
实现类FileSnap
属性
File snapDir;//snap文件目录
private volatile boolean close = false;//是否关闭
private static final int VERSION=2;
private static final long dbId=-1;
private static final Logger LOG = LoggerFactory.getLogger(FileSnap.class);
public final static int SNAP_MAGIC
= ByteBuffer.wrap("ZKSN".getBytes()).getInt();//文件的魔数
函数
反序列化涉及的函数
1. public long deserialize(DataTree dt, Map<Long, Integer> sessions)
public long deserialize(DataTree dt, Map<Long, Integer> sessions)
throws IOException {
// we run through 100 snapshots (not all of them)
// if we cannot get it running within 100 snapshots
// we should give up
//找到最新的100个(大概率)valid的快照文件
List<File> snapList = findNValidSnapshots(100);
if (snapList.size() == 0) {
return -1L;
}
File snap = null;
boolean foundValid = false;
//将这些快照文件按新旧排序，直到第一个合法的就break
for (int i = 0; i < snapList.size(); i++) {
snap = snapList.get(i);
InputStream snapIS = null;
CheckedInputStream crcIn = null;
try {
LOG.info("Reading snapshot " + snap);
snapIS = new BufferedInputStream(new FileInputStream(snap));
crcIn = new CheckedInputStream(snapIS, new Adler32());
InputArchive ia = BinaryInputArchive.getArchive(crcIn);
deserialize(dt,sessions, ia);//根据ia反序列化到dataTree以及sessions
long checkSum = crcIn.getChecksum().getValue();//反序列填充session和dataTree之后，计算checkSum
long val = ia.readLong("val");
if (val != checkSum) {//比较checkSum
throw new IOException("CRC corruption in snapshot : " + snap);
}
foundValid = true;//如果前100个有一个valid，就break
break;
} catch(IOException e) {
LOG.warn("problem reading snap file " + snap, e);
} finally {
if (snapIS != null)
snapIS.close();
if (crcIn != null)
crcIn.close();
}
}
if (!foundValid) {//如果前100个中一个valid都没有
throw new IOException("Not able to find valid snapshots in " + snapDir);
}
//从最近的第一个valid的snap文件中，解析出zxid
dt.lastProcessedZxid = Util.getZxidFromName(snap.getName(), "snapshot");
return dt.lastProcessedZxid;
}
里面调用了findNValidSnapshots函数 以及deserialize(dt,sessions, ia);
2.findNValidSnapshots函数
//找到最近n个(大概)合理的快照文件，按从新到旧排序
private List<File> findNValidSnapshots(int n) throws IOException {
List<File> files = Util.sortDataDir(snapDir.listFiles(),"snapshot", false);
int count = 0;
List<File> list = new ArrayList<File>();
for (File f : files) {
// we should catch the exceptions
// from the valid snapshot and continue
// until we find a valid one
try {
if (Util.isValidSnapshot(f)) {//一个minor check,来看这个文件是否大概率valid
list.add(f);
count++;
if (count == n) {
break;
}
}
} catch (IOException e) {
LOG.info("invalid snapshot " + f, e);
}
}
return list;
}
3.deserialize(dt,sessions, ia)函数
public void deserialize(DataTree dt, Map<Long, Integer> sessions,
InputArchive ia) throws IOException {
FileHeader header = new FileHeader();
header.deserialize(ia, "fileheader");// 反序列化至header
if (header.getMagic() != SNAP_MAGIC) {
throw new IOException("mismatching magic headers "
+ header.getMagic() +
" != " + FileSnap.SNAP_MAGIC);
}
SerializeUtils.deserializeSnapshot(dt,ia,sessions);// 反序列化至dataTree和sessions
}
序列化涉及的函数
public synchronized void serialize(DataTree dt, Map<Long, Integer> sessions, File snapShot)
public synchronized void serialize(DataTree dt, Map<Long, Integer> sessions, File snapShot)
throws IOException {
if (!close) {
OutputStream sessOS = new BufferedOutputStream(new FileOutputStream(snapShot));
CheckedOutputStream crcOut = new CheckedOutputStream(sessOS, new Adler32());
//CheckedOutputStream cout = new CheckedOutputStream()
OutputArchive oa = BinaryOutputArchive.getArchive(crcOut);
FileHeader header = new FileHeader(SNAP_MAGIC, VERSION, dbId);
serialize(dt,sessions,oa, header);//将dt,session,header进行序列化
long val = crcOut.getChecksum().getValue();
oa.writeLong(val, "val");//得到checksum，写入
oa.writeString("/", "path");//写入"\"作为结束标记,这也是判断是否highly valid的条件之一
sessOS.flush();
crcOut.close();
sessOS.close();
}
}
里面调用了serialize(dt,sessions,oa, header);函数
protected void serialize(DataTree dt,Map<Long, Integer> sessions, OutputArchive oa, FileHeader header)
protected void serialize(DataTree dt,Map<Long, Integer> sessions,
OutputArchive oa, FileHeader header) throws IOException {
// this is really a programmatic error and not something that can
// happen at runtime
if(header==null)
throw new IllegalStateException(
"Snapshot's not open for writing: uninitialized header");
header.serialize(oa, "fileheader");//header序列化
SerializeUtils.serializeSnapshot(dt,oa,sessions);//将dataTree和sessions序列化到oa
}
找到最近的snapshot文件
public File findMostRecentSnapshot() throws IOException {
List<File> files = findNValidSnapshots(1);
if (files.size() == 0) {
return null;
}
return files.get(0);
}
很好理解
关闭，释放资源
@Override
public synchronized void close() throws IOException {
close = true;
}
很好理解
回滚日志
public synchronized void rollLog() throws IOException {
if (logStream != null) {
this.logStream.flush();
this.logStream = null;
oa = null;
}
}
可视化工具
利用SnapshotFormatter.java即可
不过针对http://www.jianshu.com/p/d1f8b9d6ad57贴出的demo
看源码应该是snapshot.xxx这样的文件，但是我并没有看到生成快照文件
思考
snapshot文件，怎么样算大概率valid,大概率valid什么时候不是truely valid
org.apache.zookeeper.server.persistence.Util#isValidSnapshot中说
as in a situation when the server dies while in the process
of storing a snapshot. Any file that is not a snapshot is also
an invalid snapshot.
org.apache.zookeeper.server.persistence.FileSnap#findNValidSnapshots中说
This does not mean that the snapshot is truly valid but is valid with a high probability
补充什么时候算大概率valid但是not truly valid
org.apache.zookeeper.server.persistence.FileSnap#deserialize(org.apache.zookeeper.server.DataTree, java.util.Map<java.lang.Long,java.lang.Integer>)
函数里面检查了checkSum
也就是说snapShot文件格式之类的都ok，但是数据内容被改了导致checkSum不一致，我认为是这些情况。
上一节FileTxnLog和这一节FileSnap有什么关系
上一节FileTxnLog是事务日志文件，就是各个node的增删改等
这一节FileSnap是快照，就是内存中是以DataTree是什么样子的（可能不是最新）
后续
关于疑问
FileTxnLog和FileSnap时间先后顺序如何，两者最新的zxid是有固定的大小关系还是没有
回滚，恢复和快照有关系吗
会在下一节进行讲解
问题
什么时候生成快照，快照什么时候被删除,会不会被删除
如果zk集群挂了是从哪里恢复，FileSnap还是FileTxn，FileSnap都不一定是最新的，zxid怎么保证
refer
http://www.cnblogs.com/leesf456/p/6285014.html
http://blog.csdn.net/quhongwei_zhanqiu?viewmode=contents
http://www.cnblogs.com/leesf456/p/6179118.html
http://www.solinx.co/archives/448?utm_source=tuicool&utm_medium=referral
