Logstash sending data to kafka topic at same time which disturbs ordering of log transactions

Elastic Stack Logstash
1

Hello Everyone,

My log file transactions look like this as shown below

My log file looks like this:

start time: 2021-10-11T13:54:34Z
category: commercial
status: started
end time:

start time: 2021-10-11T13:54:34Z
category: commercial
status: In Progress
end time:

start time: 2021-10-11T13:54:34Z
category: commercial
status: Completed
end time: 2021-10-11T13:59:12Z

There are many transactions like above in a log file.

I am using logstash to push those transactions into kafka topic.

Problem:
when logstash pushed these transactions to kafka, most of the transactions are reaching kafka topic at same time and because of this, ordering of log data is getting disturbed.

Hence, both of the transactions with status "In Progress" and "Completed" reaching at same time and when I view ordering at kafka side, transaction with status "Completed" appearing before transaction with status "In Progress" as both having same time.

So, at kibana dashboard, most of the transaction status are in "In Progress" rather than "Completed".

What can I do in logstash to enforce ordering of transaction? My log file does not contain any other dates/time other than start time and end time.

In logstash.yml file, pipeline.ordered is set to "auto" and number of workers is set to default.

I am new to logstash and I am struggling to resolve this issue.

Kindly help me in fixing this problem.

2

If you have more than 1 CPU core in the logstash server, you should set pipeline.workers to 1, the default behavior is to have 1 worker per CPU core, so if you have a 4 cores machine, you will be using 4 workers.

So, to make your events ordered you need to set pipeline.workers to 1.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.