Filebeat:Events in Kibana not in Sequence

Elastic Stack Beats
1

I have configured filebeat to a kafka output which is subsequently fed to elastic and then visualized in kibana.
ApplicationLogfile-->Filebeat-->kafka-->logstash-->Kibana

The issue is, when the Application writes events in Log files the result in kibana is not in sequence.

I have tried with a fileoutput and the result in the file is perfectly in sequence.

I have the following config in filebeat.yml
output.kafka:
hosts: ["XX.XX.XXX.X:9092","XX.XX.XXX.X:9092","XX.XX.XXX.X:9092"]
topic: "XYZ-logging-XYZApplication"
required_acks: 1
output.file:
path: "/data/apps"
filename: filebeat

2

it's partially of how kafka operates. Kafka splits a topic into multiple partitions. All partitions act like concurrent queues (there is no ordering guarantees between partitions). By default the kafka output uses random partition selection, selecting a new partition for every event. You can increase the number of events being pushed to one partition, before selecting another one, but in general this behavior is due to load-balancing. Normally you should be able to sort events by timestamp and offset in kibana.

3

thanks steffens, Can I sort the table in kibana on both timestamp and offset ? I thought only one field sort was possible ?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.