Logstash: Resource consumption of multiple Logstash servers are not even

[Hung_Phan_Huy (Hùng Phan Huy)]

Hi,

I'm running my Logstash services on Docker Swarm with 4 replicas. They subscribe to the same Kafka topic and output the data to Elasticsearch.

When I inspect the resource consumption of those 4 containers, I found that the memory is even, but the CPU is not. 2 containers consume roughly 50% CPU but the other 2 never exceed 2%.

Am I having an issue with the other 2 containers with low CPU consumption?

Some useful logs in Logstash containers with low CPU consumption:

[2021-01-07T05:02:56,787][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][75103990350a1b4ea8e9c7e115f84685510dc8689748dfad19b2d52677e6a771] [Consumer clientId=logstash-0, groupId=logstash] Discovered group coordinator 10.128.151.164:9092 (id: 2147482646 rack: null),
[2021-01-07T05:02:56,790][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][75103990350a1b4ea8e9c7e115f84685510dc8689748dfad19b2d52677e6a771] [Consumer clientId=logstash-0, groupId=logstash] (Re-)joining group,
[2021-01-07T05:02:59,785][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][75103990350a1b4ea8e9c7e115f84685510dc8689748dfad19b2d52677e6a771] [Consumer clientId=logstash-0, groupId=logstash] Successfully joined group with generation 923,
[2021-01-07T05:02:59,787][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][main][75103990350a1b4ea8e9c7e115f84685510dc8689748dfad19b2d52677e6a771] [Consumer clientId=logstash-0, groupId=logstash] Adding newly assigned partitions: ,
[2021-01-07T05:03:08,789][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][75103990350a1b4ea8e9c7e115f84685510dc8689748dfad19b2d52677e6a771] [Consumer clientId=logstash-0, groupId=logstash] Attempt to heartbeat failed since group is rebalancing,
[2021-01-07T05:03:08,790][INFO ][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator][main][75103990350a1b4ea8e9c7e115f84685510dc8689748dfad19b2d52677e6a771] [Consumer clientId=logstash-0, groupId=logstash] Revoke previously assigned partitions ,
[2021-01-07T05:03:08,790][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][main][75103990350a1b4ea8e9c7e115f84685510dc8689748dfad19b2d52677e6a771] [Consumer clientId=logstash-0, groupId=logstash] (Re-)joining group,
[2021-01-07T05:02:56,786][INFO ][org.apache.kafka.clients.Metadata][main][75103990350a1b4ea8e9c7e115f84685510dc8689748dfad19b2d52677e6a771] [Consumer clientId=logstash-0, groupId=logstash] Cluster ID: BIbPq3azQnSIjx2QURxtBA

Thanks in advance.

[Hung_Phan_Huy (Hùng Phan Huy)]

The reason is that I only have 2 partitions for my Kafka topics so only 2 Logstash instances can subscribe to those 2 partitions. The other 2 are useless.