开篇
这篇文章主要用于讲解清楚Dubbo Admin的监控数据的来源，如何通过zookeeper作为注册中心来获取实际运行中的数据进行服务治理。
这篇文章主要侧重于数据方面的获取包括consumers, configurators, routers, providers，具体如何进行服务治理后面文章会具体进行分析。
文章是基于dubbo-2.6.0的版本进行分析。
注册节点
[zk] ls /dubbo/com.alibaba.dubbo.demo.DemoService
[consumers, configurators, routers, providers]
zookeeper服务树
- Dubbo的服务树如上图所示，以服务如com.foo.BarService作为第一层节点Service层。
- Service层包含Type层，包括consumers, configurators, routers, providers，监控数据主要是指Type层子节点的数据监控。
实现过程概述
1.依次按照/dubbo，/dubbo/service，/dubbo/service/type的顺序进行发现并最终订阅/dubbo/service/type的节点。
2.首次获取所有/dubbo/service/type的所有子节点并保存导数据结构registryCache当中。
3.后续/dubbo/service/type的节点变更通过zookeeper事件通知机制更新到registryCache当中。
4.registryCache保存了zookeeper上Dubbo服务节点上的所有信息，按照
ConcurrentMap<category, ConcurrentMap<servicename, Map<Long, URL>>>的数据结构进行保存，其中category包含providers,consumers,routers,configurators。
实现过程源码分析
dubbo-admin目录
dubbo-admin的目录结构如上图，RegistryServerSync作为同步数据的核心类。
RegistryServerSync的数据用于同步并监听zookeeper中关于Dubbo服务的数据。
RegistryServerSync的bean定义
<dubbo:application name="dubbo-admin"/>
<dubbo:registry address="${dubbo.registry.address}" check="false" file="false"/>
<dubbo:reference id="registryService" interface="com.alibaba.dubbo.registry.RegistryService" check="false"/>
<bean id="configService" class="com.alibaba.dubbo.governance.service.impl.ConfigServiceImpl"/>
<bean id="consumerService" class="com.alibaba.dubbo.governance.service.impl.ConsumerServiceImpl"/>
<bean id="overrideService" class="com.alibaba.dubbo.governance.service.impl.OverrideServiceImpl"/>
<bean id="ownerService" class="com.alibaba.dubbo.governance.service.impl.OwnerServiceImpl"/>
<bean id="providerService" class="com.alibaba.dubbo.governance.service.impl.ProviderServiceImpl"/>
<bean id="routeService" class="com.alibaba.dubbo.governance.service.impl.RouteServiceImpl"/>
<bean id="userService" class="com.alibaba.dubbo.governance.service.impl.UserServiceImpl">
<property name="rootPassword" value="${dubbo.admin.root.password}"/>
<property name="guestPassword" value="${dubbo.admin.guest.password}"/>
</bean>
<bean id="governanceCache" class="com.alibaba.dubbo.governance.sync.RegistryServerSync"/>
- RegistryServerSync的定义在META-INF/spring/dubbo-admin.xml当中。
RegistryServerSync初始化订阅过程
public class RegistryServerSync implements InitializingBean, DisposableBean, NotifyListener {
private static final Logger logger = LoggerFactory.getLogger(RegistryServerSync.class);
// admin://192.168.1.5?category=providers,consumers,routers,configurators&check=false&classifier=*&enabled=*&group=*&interface=*&version=*
private static final URL SUBSCRIBE = new URL(Constants.ADMIN_PROTOCOL, NetUtils.getLocalHost(), 0, "",
Constants.INTERFACE_KEY, Constants.ANY_VALUE,
Constants.GROUP_KEY, Constants.ANY_VALUE,
Constants.VERSION_KEY, Constants.ANY_VALUE,
Constants.CLASSIFIER_KEY, Constants.ANY_VALUE,
Constants.CATEGORY_KEY, Constants.PROVIDERS_CATEGORY + ","
+ Constants.CONSUMERS_CATEGORY + ","
+ Constants.ROUTERS_CATEGORY + ","
+ Constants.CONFIGURATORS_CATEGORY,
Constants.ENABLED_KEY, Constants.ANY_VALUE,
Constants.CHECK_KEY, String.valueOf(false));
private static final AtomicLong ID = new AtomicLong();
/**
* Make sure ID never changed when the same url notified many times
*/
private final ConcurrentHashMap<String, Long> URL_IDS_MAPPER = new ConcurrentHashMap<String, Long>();
// ConcurrentMap<category, ConcurrentMap<servicename, Map<Long, URL>>>
private final ConcurrentMap<String, ConcurrentMap<String, Map<Long, URL>>> registryCache = new ConcurrentHashMap<String, ConcurrentMap<String, Map<Long, URL>>>();
@Autowired
private RegistryService registryService;
public ConcurrentMap<String, ConcurrentMap<String, Map<Long, URL>>> getRegistryCache() {
return registryCache;
}
public void afterPropertiesSet() throws Exception {
logger.info("Init Dubbo Admin Sync Cache...");
registryService.subscribe(SUBSCRIBE, this);
}
}
category参数
RegistryServerSync在afterPropertiesSet()方法内部执行订阅操作。
SUBSCRIBE变量中category=providers,consumers,routers,configurators表示我们需要订阅service下的这4类节点，其中routers和configurators就是跟服务治理相关。
SUBSCRIBE的check=false&classifier=&enabled=&group=&interface=&version=*等变量都是通用符*，这里我们关注下interface=*的变量，后面执行实际订阅会根据该变量做判断。
RegistryServerSync执行订阅过程
- 参见ZookeeperRegistry#doSubscribe
protected void doSubscribe(final URL url, final NotifyListener listener) {
try {
// admin订阅过程中，首次传入参数interface的值为*，所以走的这个分支
if (Constants.ANY_VALUE.equals(url.getServiceInterface())) {
String root = toRootPath();
ConcurrentMap<NotifyListener, ChildListener> listeners = zkListeners.get(url);
if (listeners == null) {
zkListeners.putIfAbsent(url, new ConcurrentHashMap<NotifyListener, ChildListener>());
listeners = zkListeners.get(url);
}
ChildListener zkListener = listeners.get(listener);
if (zkListener == null) {
listeners.putIfAbsent(listener, new ChildListener() {
public void childChanged(String parentPath, List<String> currentChilds) {
for (String child : currentChilds) {
child = URL.decode(child);
if (!anyServices.contains(child)) {
anyServices.add(child);
subscribe(url.setPath(child).addParameters(Constants.INTERFACE_KEY, child,
Constants.CHECK_KEY, String.valueOf(false)), listener);
}
}
}
});
zkListener = listeners.get(listener);
}
zkClient.create(root, false);
// 获取 /dubbo目录的子节点，返回service层所有service接口
// 如/dubbo/com.alibaba.dubbo.demo.DemoServiceEcho
List<String> services = zkClient.addChildListener(root, zkListener);
if (services != null && services.size() > 0) {
for (String service : services) {
service = URL.decode(service);
anyServices.add(service);
// 针对每个service进行subscribe操作
subscribe(url.setPath(service).addParameters(Constants.INTERFACE_KEY, service,
Constants.CHECK_KEY, String.valueOf(false)), listener);
}
}
} else {
List<URL> urls = new ArrayList<URL>();
// 获取service下的category，consumers, configurators, routers, providers
// /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/providers
// /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/consumers
// /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/routers
// /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/configurators
for (String path : toCategoriesPath(url)) {
ConcurrentMap<NotifyListener, ChildListener> listeners = zkListeners.get(url);
if (listeners == null) {
zkListeners.putIfAbsent(url, new ConcurrentHashMap<NotifyListener, ChildListener>());
listeners = zkListeners.get(url);
}
ChildListener zkListener = listeners.get(listener);
if (zkListener == null) {
listeners.putIfAbsent(listener, new ChildListener() {
public void childChanged(String parentPath, List<String> currentChilds) {
ZookeeperRegistry.this.notify(url, listener, toUrlsWithEmpty(url, parentPath, currentChilds));
}
});
zkListener = listeners.get(listener);
}
zkClient.create(path, false);
// 针对每个service的监听以下节点
// /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/providers
// /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/consumers
// /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/routers
// /dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/configurators
List<String> children = zkClient.addChildListener(path, zkListener);
if (children != null) {
urls.addAll(toUrlsWithEmpty(url, path, children));
}
}
// urls为某个服务下所有的子节点内容
// dubbo://172.17.32.8:20880/com.alibaba.dubbo.demo.DemoServiceEcho?anyhost=true&application=demo-provider&bean.name=com.alibaba.dubbo.demo.DemoServiceEcho&dubbo=2.0.2&generic=false&interface=com.alibaba.dubbo.demo.DemoServiceEcho&methods=sayHello&pid=79153&side=provider×tamp=1577783208090
// empty://192.168.1.5/com.alibaba.dubbo.demo.DemoServiceEcho?category=consumers&check=false&classifier=*&enabled=*&group=*&interface=com.alibaba.dubbo.demo.DemoServiceEcho&version=*
// route://0.0.0.0/com.alibaba.dubbo.demo.DemoServiceEcho?category=routers&dynamic=false&enabled=true&force=true&name=com.alibaba.dubbo.demo.DemoServiceEcho blackwhitelist&priority=0&router=condition&rule=consumer.host+%3D+1.1.1.1+%3D%3E+false&runtime=false
// override://192.168.1.5/com.alibaba.dubbo.demo.DemoServiceEcho?category=configurators&dynamic=false&enabled=true&weight=12
// override://192.168.1.5/com.alibaba.dubbo.demo.DemoServiceEcho?category=configurators&dynamic=false&enabled=true&weight=12
// override://1.1.1.1/com.alibaba.dubbo.demo.DemoServiceEcho?category=configurators&dynamic=false&enabled=true&weight=13
notify(url, listener, urls);
}
} catch (Throwable e) {
throw new RpcException("Failed to subscribe " + url + " to zookeeper " + getUrl() + ", cause: " + e.getMessage(), e);
}
}
订阅的过程是一个逐层发现并订阅的过程。
获取根节点/dubbo下的所有service节点集合，如/dubbo/com.alibaba.dubbo.demo.DemoServiceEcho。
针对每个service节点获取该节点的所有子节点(描述方便用type表示)，包括providers,consumers,routers,configurators。
针对每个type如/dubbo/com.alibaba.dubbo.demo.DemoServiceEcho/providers进行监听。
首次启动会获取每个type节点下的所有子节点进行第一轮初始化过程，后续的变更都是通过type节点本身的监听回调进行实现。
首次启动会把每个service下所有的type下的所有的子节点合并成urls后进行notify动作。
RegistryServerSync回调分组过程
- 参见AbstractRegistry#notify
public abstract class AbstractRegistry implements Registry {
protected void notify(URL url, NotifyListener listener, List<URL> urls) {
if (url == null) {
throw new IllegalArgumentException("notify url == null");
}
if (listener == null) {
throw new IllegalArgumentException("notify listener == null");
}
if ((urls == null || urls.size() == 0)
&& !Constants.ANY_VALUE.equals(url.getServiceInterface())) {
logger.warn("Ignore empty notify urls for subscribe url " + url);
return;
}
if (logger.isInfoEnabled()) {
logger.info("Notify urls for subscribe url " + url + ", urls: " + urls);
}
// 根据category进行分组，包括consumers, configurators, routers, providers
Map<String, List<URL>> result = new HashMap<String, List<URL>>();
for (URL u : urls) {
if (UrlUtils.isMatch(url, u)) {
String category = u.getParameter(Constants.CATEGORY_KEY, Constants.DEFAULT_CATEGORY);
List<URL> categoryList = result.get(category);
if (categoryList == null) {
categoryList = new ArrayList<URL>();
result.put(category, categoryList);
}
categoryList.add(u);
}
}
if (result.size() == 0) {
return;
}
Map<String, List<URL>> categoryNotified = notified.get(url);
if (categoryNotified == null) {
notified.putIfAbsent(url, new ConcurrentHashMap<String, List<URL>>());
categoryNotified = notified.get(url);
}
for (Map.Entry<String, List<URL>> entry : result.entrySet()) {
String category = entry.getKey();
List<URL> categoryList = entry.getValue();
categoryNotified.put(category, categoryList);
saveProperties(url);
// 按照分组的结果进行回调通知
listener.notify(categoryList);
}
}
}
RegistryServerSync的回调过程中根据category进行分组，分组包括consumers, configurators, routers, providers。
依次针对分组后的结果进行回调通知，执行RegistryServerSync的notify动作。
RegistryServerSync保存回调结果
- 参见RegistryServerSync#notify
public class RegistryServerSync implements InitializingBean, DisposableBean, NotifyListener {
/**
* Make sure ID never changed when the same url notified many times
*/
private final ConcurrentHashMap<String, Long> URL_IDS_MAPPER = new ConcurrentHashMap<String, Long>();
// ConcurrentMap<category, ConcurrentMap<servicename, Map<Long, URL>>>
// servicename=groupName/serviceName:versionNum
private final ConcurrentMap<String, ConcurrentMap<String, Map<Long, URL>>> registryCache = new ConcurrentHashMap<String, ConcurrentMap<String, Map<Long, URL>>>();
public void notify(List<URL> urls) {
if (urls == null || urls.isEmpty()) {
return;
}
// Map<category, Map<servicename, Map<Long, URL>>>
final Map<String, Map<String, Map<Long, URL>>> categories = new HashMap<String, Map<String, Map<Long, URL>>>();
String interfaceName = null;
for (URL url : urls) {
String category = url.getParameter(Constants.CATEGORY_KEY, Constants.PROVIDERS_CATEGORY);
// 针对empty的情况，移出已经消失的服务
if (Constants.EMPTY_PROTOCOL.equalsIgnoreCase(url.getProtocol())) { // NOTE: group and version in empty protocol is *
ConcurrentMap<String, Map<Long, URL>> services = registryCache.get(category);
if (services != null) {
String group = url.getParameter(Constants.GROUP_KEY);
String version = url.getParameter(Constants.VERSION_KEY);
// NOTE: group and version in empty protocol is *
if (!Constants.ANY_VALUE.equals(group) && !Constants.ANY_VALUE.equals(version)) {
services.remove(url.getServiceKey());
} else {
for (Map.Entry<String, Map<Long, URL>> serviceEntry : services.entrySet()) {
String service = serviceEntry.getKey();
if (Tool.getInterface(service).equals(url.getServiceInterface())
&& (Constants.ANY_VALUE.equals(group) || StringUtils.isEquals(group, Tool.getGroup(service)))
&& (Constants.ANY_VALUE.equals(version) || StringUtils.isEquals(version, Tool.getVersion(service)))) {
services.remove(service);
}
}
}
}
} else {
// 添加服务到全局的registryCache变量当中
if (StringUtils.isEmpty(interfaceName)) {
interfaceName = url.getServiceInterface();
}
// 用于保存局部变量的categories
Map<String, Map<Long, URL>> services = categories.get(category);
if (services == null) {
services = new HashMap<String, Map<Long, URL>>();
categories.put(category, services);
}
// service=groupName/interfaceName:versionNum
String service = url.getServiceKey();
Map<Long, URL> ids = services.get(service);
if (ids == null) {
ids = new HashMap<Long, URL>();
services.put(service, ids);
}
// Make sure we use the same ID for the same URL
if (URL_IDS_MAPPER.containsKey(url.toFullString())) {
ids.put(URL_IDS_MAPPER.get(url.toFullString()), url);
} else {
long currentId = ID.incrementAndGet();
ids.put(currentId, url);
URL_IDS_MAPPER.putIfAbsent(url.toFullString(), currentId);
}
}
}
if (categories.size() == 0) {
return;
}
// 本次category对应的数据不为空进行添加动作
for (Map.Entry<String, Map<String, Map<Long, URL>>> categoryEntry : categories.entrySet()) {
String category = categoryEntry.getKey();
ConcurrentMap<String, Map<Long, URL>> services = registryCache.get(category);
if (services == null) {
services = new ConcurrentHashMap<String, Map<Long, URL>>();
registryCache.put(category, services);
} else {
Set<String> keys = new HashSet<String>(services.keySet());
// 移出已经不存在的数据，同一个接口但是不在本次的回调数据当中
for (String key : keys) {
if (Tool.getInterface(key).equals(interfaceName) && !categoryEntry.getValue().entrySet().contains(key)) {
services.remove(key);
}
}
}
// 用最新的数据进行覆盖
services.putAll(categoryEntry.getValue());
}
}
}
RegistryServerSync的notify()方法的urls参数是/dubbo/service/type下的子节点。
Dubbo的服务树的数据最终通过ConcurrentMap<category, ConcurrentMap<servicename, Map<Long, URL>>>的数据结构进行保存，保存的对象为registryCache，servicename=groupName/serviceName:versionNum的格式保存。
registryCache保存的Zookeeper上的Dubbo的服务节点内容。所有的服务治理操作依据的数据都在registryCache当中，任何服务节点的变更都会导致registryCache的数据更新。
registryCache是整个服务治理的数据核心。
