Ruby filter processes events in incorrect order

[Heckler_GlobalOperat (Heckler Global Operation)]

Here is my test configuration
I create a simple list of events like below, the events are numbered according to their order of appearance

2016-12-06 07:00:00,251 event1
2016-12-06 07:00:10,651 event2
2016-12-06 07:02:00,251 event3
2016-12-06 07:05:00,451 event4

In the ruby filter, I define a global variable $messageNo to keep track the number of event

ruby {
code => "
unless defined? $messageNo;
$messageNo = 0
end
$messageNo += 1;
event.set('messageNo', $messageNo);
"
}

In the end, I insert the processed events in Elasticsearch

output {
elasticsearch {
hosts => ["localhost:9200"]
index => "message_index"
}
}

But looking at the results in Elasticsearch (pictures attached), the events are not processed in order in Ruby. In the pictures, you can see 'event1' has messageNo 4 while 'event3' has messageNo 1.

[eperry (Ed)]

This is correct, Logstash filters are batch oriented and threaded, there is no way to guarantee order in which they are processed

[Heckler_GlobalOperat (Heckler Global Operation)]

This is a problem to me I must find a way to process them in sequence

[eperry (Ed)]

Why not just use the time stamp? As an order mechanism?

[Heckler_GlobalOperat (Heckler Global Operation)]

yes, the @timestamp field is in order, but I'm thinking how to control the order in which Ruby script processes those events.

[eperry (Ed)]

What are you trying to accomplish? you will have to write some other tool as Logstash is not going to be able to do what you want. I think you may have to rethink how to get what you need.

[magnusbaeck (Magnus Bäck)]

Have you tried setting the number of pipeline workers to one?

[Heckler_GlobalOperat (Heckler Global Operation)]

Thank you, it works !! Reducing the number of workers to 1 allows it to process the events in sequence
I also find a good tutorial about creating a custom filter to keep track of the event order - https://stackoverflow.com/questions/23920655/include-monotonically-increasing-value-in-logstash-field/23921517#23921517