ENV:
kafka. 2.12-2.0.1 x 3
filebeat version 7.12.1. x N
kafka config file:
broker.id=111
listeners=PLAINTEXT://172.30.160.16:9091
advertised.listeners=PLAINTEXT://172.30.160.16:9091
num.network.threads=48
num.io.threads=96
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=172.30.160.16:2181,10.110.70.10:2181,10.110.70.11:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
...
filebeat config file:
max_procs: 1
filebeat.inputs:
- type: kafka
hosts:
- 172.30.160.16:9091
- 10.110.70.10:9092
- 10.110.70.11:9093
topics: ["pre-log"]
group_id: "pre-log-consumer01"
client_id: "c01"
output.console:
pretty: true
When I started my first FileBeat consumption of Kafka messages, everything was fine.
But when I launch the second FileBeat, Kafka triggers the 'rebalancing,' which lasts for a few hours. And they can't consume it.
During this period, the first Filebeat to start is consumed normally.
Kafka's log has been outputting these logs：
...
[2021-07-02 16:40:13,276] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 907 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:13,277] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 907 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:13,281] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 907 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:16,564] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 908 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:16,565] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 908 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:16,568] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 908 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:19,849] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 909 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:19,849] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 909 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:19,852] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 909 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:23,144] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 910 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:23,145] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 910 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:23,148] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 910 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:26,433] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 911 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:26,433] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 911 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:26,437] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 911 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:29,714] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 912 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:29,715] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 912 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:29,717] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 912 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:33,003] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 913 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:33,003] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 913 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:33,006] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 913 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:36,285] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 914 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:36,285] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 914 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:36,288] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 914 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:39,579] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 915 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:39,579] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 915 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:39,582] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 915 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:42,864] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 916 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:42,864] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 916 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:42,868] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 916 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:46,152] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 917 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:46,152] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 917 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:46,155] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 917 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:49,438] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 918 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:49,439] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 918 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:49,441] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 918 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:52,725] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 919 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:52,725] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 919 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:52,728] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 919 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:56,025] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 920 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:56,026] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 920 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:56,028] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 920 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:59,322] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 921 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:59,322] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 921 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:40:59,325] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 921 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:02,606] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 922 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:02,607] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 922 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:02,609] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 922 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:05,892] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 923 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:05,892] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 923 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:05,895] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 923 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:09,179] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 924 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:09,179] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 924 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:09,182] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 924 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:12,467] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 925 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:12,467] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 925 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:12,470] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 925 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:15,761] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 926 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:15,761] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 926 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:15,764] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 926 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:19,049] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 927 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:19,050] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 927 (kafka.coordinator.group.GroupCoordinator)
[2021-07-02 16:41:19,052] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 927 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator)
...
Who can tell me what I should do about the problem that the 'rebalancing' is too long
Thanks