apollo默认支持集群部署，在apoolo中默认提供了metaservice用于服务提供服务发现，metaservice不单独部署默认集成在configservice服务中。
同时apollo提供了两个客户端:
- client：用于集成到业务应用中，提供业务应用获取配置中心的配置以及监听配置中心配置变更动态更新业务服务的配置等功能；
- portal:用于管理配置中心中各种环境的配置信息的UI操作界面服务。
client通过configService与配置中心交互，portal通过adminService管理配置中心数据，在configService与adminService集群部署的情况下，client与portal 是通过metaservice查询对应的configservice与adminservice节点通讯；
具体实现可见：
- client的com.ctrip.framework.apollo.internals.ConfigServiceLocator
- portal的com.ctrip.framework.apollo.portal.component.AdminServiceAddressLocator
metaservice 默认集成Eureka Service对外提供服务发现，接下来通过源码查看metaservice主要提供那些API，供客户端使用:
包路径
com.ctrip.framework.apollo.metaservice
节点发现API
/**
* 服务查询控制层，用于在注册中心提供adminService与configService节点的查询
*/
@RestController
@RequestMapping("/services")
public class ServiceController {
@Autowired
private DiscoveryService discoveryService;
/**
* 查询注册中心节点列表
* @return
*/
@RequestMapping("/meta")
public List<ServiceDTO> getMetaService() {
List<InstanceInfo> instances = discoveryService.getMetaServiceInstances();
List<ServiceDTO> result = instances.stream().map(new Function<InstanceInfo, ServiceDTO>() {
@Override
public ServiceDTO apply(InstanceInfo instance) {
ServiceDTO service = new ServiceDTO();
service.setAppName(instance.getAppName());
service.setInstanceId(instance.getInstanceId());
service.setHomepageUrl(instance.getHomePageUrl());
return service;
}
}).collect(Collectors.toList());
return result;
}
/**
* 查询configService服务节点列表，供client端使用
* @param appId
* @param clientIp
* @return
*/
@RequestMapping("/config")
public List<ServiceDTO> getConfigService(
@RequestParam(value = "appId", defaultValue = "") String appId,
@RequestParam(value = "ip", required = false) String clientIp) {
List<InstanceInfo> instances = discoveryService.getConfigServiceInstances();
List<ServiceDTO> result = instances.stream().map(new Function<InstanceInfo, ServiceDTO>() {
@Override
public ServiceDTO apply(InstanceInfo instance) {
ServiceDTO service = new ServiceDTO();
service.setAppName(instance.getAppName());
service.setInstanceId(instance.getInstanceId());
service.setHomepageUrl(instance.getHomePageUrl());
return service;
}
}).collect(Collectors.toList());
return result;
}
/**
* 查询adminService服务节点列表，供portal端使用
* @return
*/
@RequestMapping("/admin")
public List<ServiceDTO> getAdminService() {
List<InstanceInfo> instances = discoveryService.getAdminServiceInstances();
List<ServiceDTO> result = instances.stream().map(new Function<InstanceInfo, ServiceDTO>() {
@Override
public ServiceDTO apply(InstanceInfo instance) {
ServiceDTO service = new ServiceDTO();
service.setAppName(instance.getAppName());
service.setInstanceId(instance.getInstanceId());
service.setHomepageUrl(instance.getHomePageUrl());
return service;
}
}).collect(Collectors.toList());
return result;
}
}
ServiceController服务对外提供http方式的API，API接口如下:
- /services/config : 提供给client获取configservice服务所有节点API
- /services/admin:提供获取adminservice服务所有节点API
- /services/meta:提供获取metaservice服务所有节点API
节点发现实现
/**
* 注册中心查询服务，用于服务节点发现
*/
@Service
public class DiscoveryService {
/**
* EurekaClient 实例
*/
@Autowired
private EurekaClient eurekaClient;
/**
* 查询configService节点列表
* @return
*/
public List<InstanceInfo> getConfigServiceInstances() {
//通过eureka查询注册中心中apollo-configservice集群信息
Application application = eurekaClient.getApplication(ServiceNameConsts.APOLLO_CONFIGSERVICE);
if (application == null) {
Tracer.logEvent("Apollo.EurekaDiscovery.NotFound", ServiceNameConsts.APOLLO_CONFIGSERVICE);
}
//返回集群中所有节点信息
return application != null ? application.getInstances() : Collections.emptyList();
}
/**
* 查询注册中心服务节点列表
* @return
*/
public List<InstanceInfo> getMetaServiceInstances() {
//通过eureka查询注册中心中apollo-metaservice集群信息
Application application = eurekaClient.getApplication(ServiceNameConsts.APOLLO_METASERVICE);
if (application == null) {
Tracer.logEvent("Apollo.EurekaDiscovery.NotFound", ServiceNameConsts.APOLLO_METASERVICE);
}
//返回集群中所有节点信息
return application != null ? application.getInstances() : Collections.emptyList();
}
/**
* 查询adminService服务节点列表
* @return
*/
public List<InstanceInfo> getAdminServiceInstances() {
//通过eureka查询注册中心中apollo-adminservice集群信息
Application application = eurekaClient.getApplication(ServiceNameConsts.APOLLO_ADMINSERVICE);
if (application == null) {
Tracer.logEvent("Apollo.EurekaDiscovery.NotFound", ServiceNameConsts.APOLLO_ADMINSERVICE);
}
//返回集群中所有节点信息
return application != null ? application.getInstances() : Collections.emptyList();
}
}
DiscoveryService类持有Eureka client实例，通过EurekaClient获取对应的服务的集群信息，再通过集群信息获取节点列表返回给具体api，至此apollo的集群节点发现就完成了，至于节点变更后动态更新等功能，是通过定时轮询api来动态更新节点变化的，具体实现可见AdminServiceAddressLocator与ConfigServiceAddressLocator类
ConfigServiceAddressLocator
/**
* configService服务节点定位器，用于提供给client查询configService的所有节点
*/
public class ConfigServiceLocator {
private static final Logger logger = LoggerFactory.getLogger(ConfigServiceLocator.class);
/**
* http通讯工具
*/
private HttpUtil m_httpUtil;
/**
* 配置工具类
*/
private ConfigUtil m_configUtil;
/**
* configService服务节点集合
*/
private AtomicReference<List<ServiceDTO>> m_configServices;
private Type m_responseType;
/**
* 定时任务线程池，用于定时刷新服务节点信息
*/
private ScheduledExecutorService m_executorService;
private static final Joiner.MapJoiner MAP_JOINER = Joiner.on("&").withKeyValueSeparator("=");
private static final Escaper queryParamEscaper = UrlEscapers.urlFormParameterEscaper();
/**
* Create a config service locator.
*/
public ConfigServiceLocator() {
List<ServiceDTO> initial = Lists.newArrayList();
m_configServices = new AtomicReference<>(initial);
m_responseType = new TypeToken<List<ServiceDTO>>() {
}.getType();
m_httpUtil = ApolloInjector.getInstance(HttpUtil.class);
m_configUtil = ApolloInjector.getInstance(ConfigUtil.class);
this.m_executorService = Executors.newScheduledThreadPool(1,
ApolloThreadFactory.create("ConfigServiceLocator", true));
//初始化加载configService列表
initConfigServices();
}
private void initConfigServices() {
//获取自定义配置的服务列表
// get from run time configurations
List<ServiceDTO> customizedConfigServices = getCustomizedConfigService();
if (customizedConfigServices != null) {
setConfigServices(customizedConfigServices);
return;
}
//更新服务列表通过metaservice
// update from meta service
this.tryUpdateConfigServices();
//启动定时任务定时请求metaservice更新服务列表
this.schedulePeriodicRefresh();
}
/**
* 获取自定义的服务列表，未通过metaservice获取
* @return
*/
private List<ServiceDTO> getCustomizedConfigService() {
//首先通过System.getProperty获取服务列表
// 1. Get from System Property
String configServices = System.getProperty("apollo.configService");
if (Strings.isNullOrEmpty(configServices)) {
//System.getenv方式获取服务列表次之
// 2. Get from OS environment variable
configServices = System.getenv("APOLLO_CONFIGSERVICE");
}
if (Strings.isNullOrEmpty(configServices)) {
//server.properties方式获取服务列表
// 3. Get from server.properties
configServices = Foundation.server().getProperty("apollo.configService", null);
}
if (Strings.isNullOrEmpty(configServices)) {
return null;
}
logger.warn("Located config services from apollo.configService configuration: {}, will not refresh config services from remote meta service!", configServices);
// mock service dto list
String[] configServiceUrls = configServices.split(",");
List<ServiceDTO> serviceDTOS = Lists.newArrayList();
for (String configServiceUrl : configServiceUrls) {
configServiceUrl = configServiceUrl.trim();
ServiceDTO serviceDTO = new ServiceDTO();
serviceDTO.setHomepageUrl(configServiceUrl);
serviceDTO.setAppName(ServiceNameConsts.APOLLO_CONFIGSERVICE);
serviceDTO.setInstanceId(configServiceUrl);
serviceDTOS.add(serviceDTO);
}
return serviceDTOS;
}
/**
* Get the config service info from remote meta server.
*
* @return the services dto
*/
public List<ServiceDTO> getConfigServices() {
if (m_configServices.get().isEmpty()) {
updateConfigServices();
}
return m_configServices.get();
}
private boolean tryUpdateConfigServices() {
try {
updateConfigServices();
return true;
} catch (Throwable ex) {
//ignore
}
return false;
}
private void schedulePeriodicRefresh() {
//定时任务定时更新服务列表
this.m_executorService.scheduleAtFixedRate(
new Runnable() {
@Override
public void run() {
logger.debug("refresh config services");
Tracer.logEvent("Apollo.MetaService", "periodicRefresh");
tryUpdateConfigServices();
}
}, m_configUtil.getRefreshInterval(), m_configUtil.getRefreshInterval(),
m_configUtil.getRefreshIntervalTimeUnit());
}
/**
* 通过调用metaservice的/services/config 定时更新服务列表
*/
private synchronized void updateConfigServices() {
//构建请求URL
String url = assembleMetaServiceUrl();
HttpRequest request = new HttpRequest(url);
int maxRetries = 2;
Throwable exception = null;
for (int i = 0; i < maxRetries; i++) {
Transaction transaction = Tracer.newTransaction("Apollo.MetaService", "getConfigService");
transaction.addData("Url", url);
try {
HttpResponse<List<ServiceDTO>> response = m_httpUtil.doGet(request, m_responseType);
transaction.setStatus(Transaction.SUCCESS);
List<ServiceDTO> services = response.getBody();
if (services == null || services.isEmpty()) {
logConfigService("Empty response!");
continue;
}
setConfigServices(services);
return;
} catch (Throwable ex) {
Tracer.logEvent("ApolloConfigException", ExceptionUtil.getDetailMessage(ex));
transaction.setStatus(ex);
exception = ex;
} finally {
transaction.complete();
}
try {
m_configUtil.getOnErrorRetryIntervalTimeUnit().sleep(m_configUtil.getOnErrorRetryInterval());
} catch (InterruptedException ex) {
//ignore
}
}
throw new ApolloConfigException(
String.format("Get config services failed from %s", url), exception);
}
/**
* 设置服务列表到本地缓存
* @param services
*/
private void setConfigServices(List<ServiceDTO> services) {
m_configServices.set(services);
logConfigServices(services);
}
/**
* 构建请求metaservice服务URL
* @return
*/
private String assembleMetaServiceUrl() {
String domainName = m_configUtil.getMetaServerDomainName();
String appId = m_configUtil.getAppId();
String localIp = m_configUtil.getLocalIp();
Map<String, String> queryParams = Maps.newHashMap();
queryParams.put("appId", queryParamEscaper.escape(appId));
if (!Strings.isNullOrEmpty(localIp)) {
queryParams.put("ip", queryParamEscaper.escape(localIp));
}
return domainName + "/services/config?" + MAP_JOINER.join(queryParams);
}
}
ConfigServiceLocator
- 通过配置获取自定义的configService列表
- 通过metaservice更新服务列表
- 创建定时任务定时通过metaservice更新服务列表
AdminServiceAddressLocator
/**
* admin service 服务URL定位器
*/
@Component
public class AdminServiceAddressLocator {
private static final long NORMAL_REFRESH_INTERVAL = 5 * 60 * 1000;
private static final long OFFLINE_REFRESH_INTERVAL = 10 * 1000;
private static final int RETRY_TIMES = 3;
private static final String ADMIN_SERVICE_URL_PATH = "/services/admin";
private static final Logger logger = LoggerFactory.getLogger(AdminServiceAddressLocator.class);
private ScheduledExecutorService refreshServiceAddressService;
private RestTemplate restTemplate;
private List<Env> allEnvs;
private Map<Env, List<ServiceDTO>> cache = new ConcurrentHashMap<>();
@Autowired
private HttpMessageConverters httpMessageConverters;
@Autowired
private PortalSettings portalSettings;
@Autowired
private RestTemplateFactory restTemplateFactory;
@PostConstruct
public void init() {
allEnvs = portalSettings.getAllEnvs();
//init restTemplate
restTemplate = restTemplateFactory.getObject();
refreshServiceAddressService =
Executors.newScheduledThreadPool(1, ApolloThreadFactory.create("ServiceLocator", true));
refreshServiceAddressService.schedule(new RefreshAdminServerAddressTask(), 1, TimeUnit.MILLISECONDS);
}
public List<ServiceDTO> getServiceList(Env env) {
List<ServiceDTO> services = cache.get(env);
if (CollectionUtils.isEmpty(services)) {
return Collections.emptyList();
}
List<ServiceDTO> randomConfigServices = Lists.newArrayList(services);
Collections.shuffle(randomConfigServices);
return randomConfigServices;
}
/**
* 刷新admin service 服务url地址任务
*/
//maintain admin server address
private class RefreshAdminServerAddressTask implements Runnable {
@Override
public void run() {
boolean refreshSuccess = true;
//遍历环境刷新服务
//refresh fail if get any env address fail
for (Env env : allEnvs) {
boolean currentEnvRefreshResult = refreshServerAddressCache(env);
refreshSuccess = refreshSuccess && currentEnvRefreshResult;
}
if (refreshSuccess) {
refreshServiceAddressService
.schedule(new RefreshAdminServerAddressTask(), NORMAL_REFRESH_INTERVAL, TimeUnit.MILLISECONDS);
} else {
refreshServiceAddressService
.schedule(new RefreshAdminServerAddressTask(), OFFLINE_REFRESH_INTERVAL, TimeUnit.MILLISECONDS);
}
}
}
/**
* 根据环境信息刷新admin service 服务URL
* @param env
* @return
*/
private boolean refreshServerAddressCache(Env env) {
for (int i = 0; i < RETRY_TIMES; i++) {
try {
ServiceDTO[] services = getAdminServerAddress(env);
if (services == null || services.length == 0) {
continue;
}
cache.put(env, Arrays.asList(services));
return true;
} catch (Throwable e) {
logger.error(String.format("Get admin server address from meta server failed. env: %s, meta server address:%s",
env, MetaDomainConsts.getDomain(env)), e);
Tracer
.logError(String.format("Get admin server address from meta server failed. env: %s, meta server address:%s",
env, MetaDomainConsts.getDomain(env)), e);
}
}
return false;
}
/**
* 根据环境查询admin service列表
* @param env
* @return
*/
private ServiceDTO[] getAdminServerAddress(Env env) {
//获取当前环境注册发现中心地址
String domainName = MetaDomainConsts.getDomain(env);
//构建查询admin service列表的请求URL
String url = domainName + ADMIN_SERVICE_URL_PATH;
//发送请求获取admin service 服务列表
return restTemplate.getForObject(url, ServiceDTO[].class);
}
}
AdminServiceAddressLocator完成的功能基本与ConfigServiceAddressLocator一致，唯一不同的是AdminServiceAddressLocator需要根据环境信息去查询对应的AdminService集群列表，这是因为portal是可以管理不同环境的配置数据的，所以需要与环境对应的adminservice来交互，而client是集成在应用中，当应用部署后，环境是不可变的，所以只需找到当前应用所对应的环境的configService交互就好。
