自定义类加载器导致的MetaspaceOOM

问题描述

在提交flinkStreamSQL时,抛出异常信息java.lang.OutOfMemoryError: Metaspace,(已设置-XX:MaxMetaspaceSize=400M 不让其动态增长)很明显元数据溢出,猜测是加载的类过多。通过获取堆转存快照发现,堆被大量DtClassLoader对象占据,这才意识到我们没有对DtClassLoader做相关缓存处理,导致每次提交任务都会创建出DtClassLoader对象,并由该对象加载我们的插件类。

使用ClassLoaderManager统一管理
  • 定义函数接口,由指定的ClassLoader去加载插件。
@FunctionalInterface
public interface ClassLoaderSupplier<T> {
    /**
     * Gets a result.
     *
     * @return a result
     */
    T get(ClassLoader cl) throws Exception;
}
  • 使用Map集合缓存类加载器对象,key为要加载插件的URL。
private static Map<String, DtClassLoader> pluginClassLoader = new ConcurrentHashMap<>();

public static <R> R newInstance(String pluginJarPath, ClassLoaderSupplier<R> supplier) throws Exception {
    ClassLoader classLoader = retrieveClassLoad(pluginJarPath);
    return ClassLoaderSupplierCallBack.callbackAndReset(supplier, classLoader);
}
//  get or create 
private static DtClassLoader retrieveClassLoad(String pluginJarPath) {
    return pluginClassLoader.computeIfAbsent(pluginJarPath, k -> {
        try {
            URL[] urls = PluginUtil.getPluginJarUrls(pluginJarPath);
            ClassLoader parentClassLoader = Thread.currentThread().getContextClassLoader();
            DtClassLoader classLoader = new DtClassLoader(urls, parentClassLoader);
            LOG.info("pluginJarPath:{} create ClassLoad successful...", pluginJarPath);
            return classLoader;
        } catch (Throwable e) {
            LOG.error("retrieve ClassLoad happens error:{}", e);
            throw new RuntimeException("retrieve ClassLoad happens error");
        }
    });
}
  • 使用集合中缓存的类加载器,回调函数接口方法加载插件,并切换回父类加载器。
public class ClassLoaderSupplierCallBack {
    public static <R> R callbackAndReset(ClassLoaderSupplier<R> supplier, ClassLoader toSetClassLoader) throws Exception {
        ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
        Thread.currentThread().setContextClassLoader(toSetClassLoader);
        try {
            return supplier.get(toSetClassLoader);
        } finally {
            Thread.currentThread().setContextClassLoader(oldClassLoader);
        }
    }
}
  • 插件调用方法调整
    转换之前,设置DtClassLoader为当前类加载器。
ClassLoader threadClassLoader = Thread.currentThread().getContextClassLoader();
    DtClassLoader parentClassloader = new DtClassLoader(new URL[]{}, threadClassLoader);
    Thread.currentThread().setContextClassLoader(parentClassloader);
    
private static AsyncReqRow loadAsyncReq(String sideType, String sqlRootDir, RowTypeInfo rowTypeInfo,
                                        JoinInfo joinInfo, List<FieldInfo> outFieldInfoList, SideTableInfo sideTableInfo) throws Exception {
    ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
    String pathOfType = String.format(PATH_FORMAT, sideType);
    String pluginJarPath = PluginUtil.getJarFileDirPath(pathOfType, sqlRootDir);
    DtClassLoader dtClassLoader = (DtClassLoader) classLoader;
    PluginUtil.addPluginJar(pluginJarPath, dtClassLoader);
    String className = PluginUtil.getSqlSideClassName(sideType, "side", OPERATOR_TYPE);
    return dtClassLoader.loadClass(className).asSubclass(AsyncReqRow.class)
            .getConstructor(RowTypeInfo.class, JoinInfo.class, List.class, SideTableInfo.class).newInstance(rowTypeInfo, joinInfo, outFieldInfoList, sideTableInfo);
}

转换后

private static AsyncReqRow loadAsyncReq(String sideType, String sqlRootDir, RowTypeInfo rowTypeInfo,
                                        JoinInfo joinInfo, List<FieldInfo> outFieldInfoList, SideTableInfo sideTableInfo) throws Exception {
    String pathOfType = String.format(PATH_FORMAT, sideType);
    String pluginJarPath = PluginUtil.getJarFileDirPath(pathOfType, sqlRootDir);
    String className = PluginUtil.getSqlSideClassName(sideType, "side", OPERATOR_TYPE);
    return ClassLoaderManager.newInstance(pluginJarPath, (cl) ->
            cl.loadClass(className).asSubclass(AsyncReqRow.class)
                    .getConstructor(RowTypeInfo.class, JoinInfo.class, List.class, SideTableInfo.class)
                    .newInstance(rowTypeInfo, joinInfo, outFieldInfoList, sideTableInfo));
} 

推荐阅读更多精彩内容

  • Swift1> Swift和OC的区别1.1> Swift没有地址/指针的概念1.2> 泛型1.3> 类型严谨 对...
    cosWriter阅读 8,452评论 1 27
  • 1. 简介 1.1 什么是 MyBatis ? MyBatis 是支持定制化 SQL、存储过程以及高级映射的优秀的...
    笨鸟慢飞阅读 3,427评论 0 4
  • 在过去(当自定义类加载器使用不普遍的时候),类几乎是“静态的”并且很少被卸载和回收,因此类也可以被看成“永久的”。...
    云狗狗狗狗狗阅读 3,850评论 0 8
  • 茶话会开始! 诗词、歌词接龙大冒险 规则: 由主持人开头,出一句诗词或一句歌词,然后按照抢答的方式接下一句,诗词或...
    灵川l阅读 49评论 0 2
  • Coco导师能量卡 《快乐》 快乐或不快乐并没有外在的原因, 那些只是借口, 我们会逐渐了解它是某种内在的状态, ...
    旺旺沫初阅读 36评论 0 1