Filebeat:Events in Kibana not in Sequence

I have configured filebeat to a kafka output which is subsequently fed to elastic and then visualized in kibana.

The issue is, when the Application writes events in Log files the result in kibana is not in sequence.

I have tried with a fileoutput and the result in the file is perfectly in sequence.

I have the following config in filebeat.yml
hosts: ["XX.XX.XXX.X:9092","XX.XX.XXX.X:9092","XX.XX.XXX.X:9092"]
topic: "XYZ-logging-XYZApplication"
required_acks: 1
path: "/data/apps"
filename: filebeat

it's partially of how kafka operates. Kafka splits a topic into multiple partitions. All partitions act like concurrent queues (there is no ordering guarantees between partitions). By default the kafka output uses random partition selection, selecting a new partition for every event. You can increase the number of events being pushed to one partition, before selecting another one, but in general this behavior is due to load-balancing. Normally you should be able to sort events by timestamp and offset in kibana.

thanks steffens, Can I sort the table in kibana on both timestamp and offset ? I thought only one field sort was possible ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.