Flink中的类加载机制
- Flink中的类加载机制
-
- Flink中的类加载配置项
- 类图关系
- parent-first类加载
- child-first类加载
- 如何处理类冲突呢
Flink中的类加载机制
Flink中的类加载配置项
Flink中关于类加载有以下配置选项
| 配置项 | 默认值 | 说明 |
|---|---|---|
| classloader.resolve-order | “child-first” | 类加载顺序。child-first优先从Flink任务(jar包)中加载类,parent-first优先从Flink集群加载类。 |
| classloader.parent-first-patterns.default | “java.; scala.; org.apache.flink.; com.esotericsoftware.kryo; org.apache.hadoop.;javax.annotation.; org.slf4j;org.apache.log4j; org.apache.logging; org.apache.commons.logging; ch.qos.logback; org.xml; javax.xml; org.apache.xerces; org.w3c” | 优先从Flink集群加载的类,以分号分隔的类前缀匹配 |
| classloader.parent-first-patterns.additional | (none) | 额外需要优先从Flink集群加载的类 |
| classloader.fail-on-metaspace-oom-error | true | 如果尝试加载用户代码类时抛出“OutOfMemoryError:Metaspace”,则Flink JVM进程失败。 |
类图关系

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | /** * This class loader accepts a custom handler if an exception occurs in {@link #loadClass(String, boolean)}. */ public abstract class FlinkUserCodeClassLoader extends URLClassLoader {<!-- --> // 加载类出现异常时的handler,默认不作处理 public static final Consumer<Throwable> NOOP_EXCEPTION_HANDLER = classLoadingException -> {<!-- -->}; // 指定一个类加载异常处理handler private final Consumer<Throwable> classLoadingExceptionHandler; protected FlinkUserCodeClassLoader(URL[] urls, ClassLoader parent) {<!-- --> this(urls, parent, NOOP_EXCEPTION_HANDLER); } protected FlinkUserCodeClassLoader( URL[] urls, ClassLoader parent, Consumer<Throwable> classLoadingExceptionHandler) {<!-- --> super(urls, parent); this.classLoadingExceptionHandler = classLoadingExceptionHandler; } /** * 实现FlinkUserCodeClassLoader的子类不需要重写loadClass方法,只需要重写loadClassWithoutExceptionHandling方法。 * loadClassWithoutExceptionHandling只负责类加载的具体实现, * 加载类时出现的异常会被捕获并交由类加载异常处理handler进行处理,并再次向上抛 * */ @Override protected final Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {<!-- --> try {<!-- --> return loadClassWithoutExceptionHandling(name, resolve); } catch (Throwable classLoadingException) {<!-- --> classLoadingExceptionHandler.accept(classLoadingException); throw classLoadingException; } } /** * Same as {@link #loadClass(String, boolean)} but without exception handling. * * <p><center>[wp_ad_camp_2]</center></p><p>Extending concrete class loaders should implement this instead of {@link #loadClass(String, boolean)}. */ protected Class<?> loadClassWithoutExceptionHandling(String name, boolean resolve) throws ClassNotFoundException {<!-- --> return super.loadClass(name, resolve); } } |
parent-first类加载
1 2 3 4 5 6 7 8 9 | /** * Regular URLClassLoader that first loads from the parent and only after that from the URLs. */ static class ParentFirstClassLoader extends FlinkUserCodeClassLoader {<!-- --> ParentFirstClassLoader(URL[] urls, ClassLoader parent, Consumer<Throwable> classLoadingExceptionHandler) {<!-- --> super(urls, parent, classLoadingExceptionHandler); } } |
那么,ParentFirstClassLoader是如何保证同一个Flink的类,优先加载集群的而非用户jar包中的?其实这里是利用Java的类加载机制——双亲委派模型。

我们知道用户提交的Flink任务运行于TaskManager进程内,TaskManager运行时的类加载器是Application ClassLoader(用户程序运行时的默认类加载器),当我们使用自定义类加载器加载需要的类时,会自底向上逐一查找,如果
child-first类加载
接着我们来看
不少人应该都用过Tomcat,Tomcat中可以部署多个Java程序,每个Java程序都包含了各自不同版本的jar包,但是他们却能各自正常的工作。同样地,同一个Flink集群也需要运行各种各样的Flink任务,而且如果用户使用的某个jar和Flink集群的jar版本不一致,需要优先加载用户的jar怎么办?Flink的
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | /** * A variant of the URLClassLoader that first loads from the URLs and only after that from the parent. * * <p>{@link #getResourceAsStream(String)} uses {@link #getResource(String)} internally so we * don't override that. */ public final class ChildFirstClassLoader extends FlinkUserCodeClassLoader {<!-- --> /** * 总是从父加载器加载的类,如以org.apache.flink开头的类等,这样所有任务都使用同一版本的link基础库 * The classes that should always go through the parent ClassLoader. This is relevant * for Flink classes, for example, to avoid loading Flink classes that cross the * user-code/system-code barrier in the user-code ClassLoader. */ private final String[] alwaysParentFirstPatterns; /** 构造方法省略 **/ @Override protected synchronized Class<?> loadClassWithoutExceptionHandling( String name, boolean resolve) throws ClassNotFoundException {<!-- --> // 如果发现类已经加载过,就不再重复加载 // First, check if the class has already been loaded Class<?> c = findLoadedClass(name); if (c == null) {<!-- --> // 总是从父加载器加载的类交由父加载器加载,ChildFirstClassLoader跳过加载这些类 // check whether the class should go parent-first for (String alwaysParentFirstPattern : alwaysParentFirstPatterns) {<!-- --> if (name.startsWith(alwaysParentFirstPattern)) {<!-- --> return super.loadClassWithoutExceptionHandling(name, resolve); } } try {<!-- --> // 查找到类之后直接返回 // check the URLs c = findClass(name); } catch (ClassNotFoundException e) {<!-- --> // let URLClassLoader do it, which will eventually call the parent // 如果当前类加载其中找不到这个类那只能交由父类加载器从Flink集群环境中查找了 c = super.loadClassWithoutExceptionHandling(name, resolve); } } if (resolve) {<!-- --> resolveClass(c); } return c; } /** 下面两个方法都是优先从子类中获取资源文件 **/ @Override public URL getResource(String name) {<!-- --> ... } @Override public Enumeration<URL> getResources(String name) throws IOException {<!-- --> ... } } |
这里
如何处理类冲突呢
上面介绍了Flink中的类加载机制,实际开发中可能会因为类加载出现ClassNotFound或者NoSuchMethodError和NoSuchFieldError等异常。这种情况可能是因为你的jar包中的版本和集群的版本不一致,优先加载了集群的class。在不改动集群默认配置的情况下,同时也防止对其他任务的影响,可以考虑对项目中引入的冲突类作shade。如
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <relocations> <relocation> <pattern>com.typesafe</pattern> <shadedPattern>com.mycompany.com.typesafe</shadedPattern> </relocation> </relocations> </configuration> </execution> </executions> </plugin> |
项目打包后,就会自动把引入和源码中相关的