docker-compose安装kafka集群


编写 docker-compose.yml 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
version: '3.2'
services:
  kafka:
    image: wurstmeister/kafka:2.12-2.4.1
    container_name: kafka          # 容器名称
    network_mode: host              # 主机模式
    restart: always                 # 失败自动重启策略
    environment:
      # 集群唯一
      - KAFKA_BROKER_ID=1
      # 对外暴露的服务端口，真正建立连接用的是KAFKA_ADVERTISED_LISTENERS
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://10.121.138.155:9092
      - KAFKA_ADVERTISED_HOST_NAME=10.121.138.155
      # 数据目录
      - KAFKA_LOG_DIRS=/kafka/data
      # 日志保留小时数
      - KAFKA_LOG_RETENTION_HOURS=168
      # zookeeper集群地址
      - KAFKA_ZOOKEEPER_CONNECT=10.121.138.155:2181,10.121.138.156:2181,10.121.138.157:2181
      # 端口
      - KAFKA_PORT=9092
      # 堆大小
      - KAFKA_HEAP_OPTS=-Xmx1G -Xms1G
      #- KAFKA_OPTS=-Djava.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf
      # 开启JMX
      - JMX_PORT=9988
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/lenovo/data/kafka/data:/kafka/data

  • KAFKA_BROKER_ID 要保证集群内唯一，部署前要修改此值
  • KAFKA_ADVERTISED_LISTENERS 中的ip地址为本机ip地址，部署前要修改此值
  • KAFKA_ADVERTISED_HOST_NAME中的iip地址为本机ip，部署前要修改此值
  • KAFKA_ZOOKEEPER_CONNECT 为zookeeper集群的ip和端口，部署前要修改此值
  • JMX_PORT 标识开启JMX。开启时要配置/etc/hosts，增加本机hostname与本机ip的对应关系，否则会报错：Error: Exception thrown by the agent : java.net.MalformedURLException: Local host name unknown: java.net.UnknownHostException: SRV-APP1: SRV-APP1: System error。如本实例中/etc/hosts要增加10.121.138.155 SRV-APP1

部署

进入到docker-compose.yml 文件所在目录，执行如下命令

1
docker-compose up -d

开启JMX_PORT导致的问题

开启JMX_PORT后，在使用kafka命令行工具（kafka-topics、kafka-console-consumer.sh等）时，会报端口被占用的异常，如：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 9999; nested exception is:
        java.net.BindException: Address in use (Bind failed)
sun.management.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 9999; nested exception is:
        java.net.BindException: Address in use (Bind failed)
        at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:480)
        at sun.management.Agent.startAgent(Agent.java:262)
        at sun.management.Agent.startAgent(Agent.java:452)
Caused by: java.rmi.server.ExportException: Port already in use: 9999; nested exception is:
        java.net.BindException: Address in use (Bind failed)
        at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:346)
        at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:254)
        at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
        at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
        at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:237)
        at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:213)
        at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:173)
        at sun.management.jmxremote.SingleEntryRegistry.<init>(SingleEntryRegistry.java:49)
        at sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:816)
        at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:468)
        ... 2 more
Caused by: java.net.BindException: Address in use (Bind failed)
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
        at java.net.ServerSocket.bind(ServerSocket.java:375)
        at java.net.ServerSocket.<init>(ServerSocket.java:237)
        at java.net.ServerSocket.<init>(ServerSocket.java:128)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createServerSocket(RMIDirectSocketFactory.java:45)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createServerSocket(RMIMasterSocketFactory.java:345)
        at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
        at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
        ... 11 more

  • 解决方式：
    修改bin/kafka-run-class.sh文件：
    把以下内容
1
2
3
if [  $JMX_PORT ]; then
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi

修改为：

1
2
3
4
5
6
7
8
9
# need to check if called to start server or client
# in order to correctly decide about JMX_PORT
ISKAFKASERVER="false"
if [[ "$*" =~ "kafka.Kafka" ]]; then
    ISKAFKASERVER="true"
fi
if [  $JMX_PORT ] && [ -z "$ISKAFKASERVER" ]; then
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi

然后重启kafka即可。

参考：https://github.com/apache/kafka/pull/1983/commits/2c5d40e946bcc149b1a9b2c01eced4ae47a734c5