# FileBeat consumes Kafka, every time a FileBeat is started as a consumer, Kafka needs to spend several hours for the rebalancing, and only the first FileBeat will not have the rebalancing **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 2, 2021, 8:56am UTC](https://discuss.elastic.co/t/filebeat-consumes-kafka-every-time-a-filebeat-is-started-as-a-consumer-kafka-needs-to-spend-several-hours-for-the-rebalancing-and-only-the-first-filebeat-will-not-have-the-rebalancing/277627 "2021-07-02T08:56:43Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![hadogen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hadogen/32/91070_2.png) [@hadogen](https://discuss.elastic.co/u/hadogen)\ **Post date:** [July 2, 2021, 8:56am UTC](https://discuss.elastic.co/t/filebeat-consumes-kafka-every-time-a-filebeat-is-started-as-a-consumer-kafka-needs-to-spend-several-hours-for-the-rebalancing-and-only-the-first-filebeat-will-not-have-the-rebalancing/277627/1 "2021-07-02T08:56:43Z")
ENV: kafka. 2.12-2.0.1 x 3 filebeat version 7.12.1. x N kafka config file: ```auto broker.id=111 listeners=PLAINTEXT://172.30.160.16:9091 advertised.listeners=PLAINTEXT://172.30.160.16:9091 num.network.threads=48 num.io.threads=96 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/data/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.flush.interval.messages=10000 log.flush.interval.ms=1000 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=172.30.160.16:2181,10.110.70.10:2181,10.110.70.11:2181 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 ``` ... filebeat config file: ```auto max_procs: 1 filebeat.inputs: - type: kafka hosts: - 172.30.160.16:9091 - 10.110.70.10:9092 - 10.110.70.11:9093 topics: ["pre-log"] group_id: "pre-log-consumer01" client_id: "c01" output.console: pretty: true ``` When I started my first FileBeat consumption of Kafka messages, everything was fine. But when I launch the second FileBeat, Kafka triggers the 'rebalancing,' which lasts for a few hours. And they can't consume it. During this period, the first Filebeat to start is consumed normally. Kafka's log has been outputting these logs： ```auto ... [2021-07-02 16:40:13,276] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 907 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:13,277] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 907 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:13,281] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 907 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:16,564] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 908 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:16,565] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 908 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:16,568] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 908 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:19,849] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 909 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:19,849] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 909 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:19,852] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 909 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:23,144] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 910 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:23,145] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 910 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:23,148] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 910 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:26,433] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 911 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:26,433] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 911 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:26,437] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 911 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:29,714] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 912 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:29,715] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 912 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:29,717] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 912 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:33,003] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 913 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:33,003] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 913 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:33,006] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 913 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:36,285] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 914 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:36,285] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 914 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:36,288] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 914 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:39,579] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 915 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:39,579] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 915 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:39,582] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 915 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:42,864] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 916 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:42,864] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 916 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:42,868] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 916 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:46,152] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 917 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:46,152] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 917 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:46,155] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 917 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:49,438] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 918 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:49,439] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 918 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:49,441] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 918 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:52,725] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 919 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:52,725] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 919 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:52,728] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 919 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:56,025] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 920 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:56,026] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 920 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:56,028] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 920 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:59,322] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 921 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:59,322] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 921 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:40:59,325] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 921 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:02,606] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 922 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:02,607] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 922 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:02,609] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 922 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:05,892] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 923 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:05,892] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 923 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:05,895] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 923 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:09,179] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 924 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:09,179] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 924 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:09,182] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 924 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:12,467] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 925 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:12,467] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 925 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:12,470] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 925 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:15,761] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 926 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:15,761] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 926 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:15,764] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 926 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:19,049] INFO [GroupCoordinator 111]: Stabilized group pre-log-consumer01 generation 927 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:19,050] INFO [GroupCoordinator 111]: Assignment received from leader for group pre-log-consumer01 for generation 927 (kafka.coordinator.group.GroupCoordinator) [2021-07-02 16:41:19,052] INFO [GroupCoordinator 111]: Preparing to rebalance group pre-log-consumer01 with old generation 927 (__consumer_offsets-26) (kafka.coordinator.group.GroupCoordinator) ... ``` Who can tell me what I should do about the problem that the 'rebalancing' is too long Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 30, 2021, 10:57am UTC](https://discuss.elastic.co/t/filebeat-consumes-kafka-every-time-a-filebeat-is-started-as-a-consumer-kafka-needs-to-spend-several-hours-for-the-rebalancing-and-only-the-first-filebeat-will-not-have-the-rebalancing/277627/2 "2021-07-30T10:57:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.