# Aggregate messages in sequence order? **URL:** \ **Category:** Logstash\ **Created:** [February 20, 2020, 6:52pm UTC](https://discuss.elastic.co/t/aggregate-messages-in-sequence-order/220251 "2020-02-20T18:52:28Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jfro](https://avatars.discourse-cdn.com/v4/letter/j/e95f7d/32.png) [@jfro](https://discuss.elastic.co/u/jfro)\ **Post date:** [February 20, 2020, 6:52pm UTC](https://discuss.elastic.co/t/aggregate-messages-in-sequence-order/220251/1 "2020-02-20T18:52:29Z")
I have vendor appliance that I'm sending syslog data from to logstash v6.5.4. The appliance splits messages that are over 1kB into separate messages and then sends them out via UDP. We had kafka buffering messages before logstash but removed that for troubleshooting. I was able to get message aggregation to mostly work using the aggregation filter however the way messages are split, it will just cut off field names and continue on in the next message (after the header). In each message I have a sequence ID (a sequential ID number that gets reset whenever), a segment total and segment number. I've used those 3 numbers to create a task\_id that's unique to that message group. The problem is message can get out of order somehow which causes issues with field names not being recombined right because for example message 5/6 could end like "permiss" and message 6/6 will have (after the header) "ions=...." This would lead to a mess of field names if allowed to continue because the break can come anywhere in a field name depending on the data. How can I use the aggregation filter (or anything else) to reassemble those messages in the order they should be based off the sequence number? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 21, 2020, 2:34am UTC](https://discuss.elastic.co/t/aggregate-messages-in-sequence-order/220251/2 "2020-02-21T02:34:14Z")
> [@jfro](#): > > The problem is message can get out of order logstash generally does not preserve the order of messages (it is multithreaded, and different subsets of messages are processed by different threads). You can set pipeline.workers to 1, so that only one CPU is used, and, for now, you will also need to disable the java\_execution engine (although a fix for that has been committed). ---
**Author:** ![jfro](https://avatars.discourse-cdn.com/v4/letter/j/e95f7d/32.png) [@jfro](https://discuss.elastic.co/u/jfro)\ **Post date:** [February 21, 2020, 1:09pm UTC](https://discuss.elastic.co/t/aggregate-messages-in-sequence-order/220251/3 "2020-02-21T13:09:46Z")
Thanks Badger. We already set the pipeline.workers=1 and we still were having issues but I don't think we tried disabling the java\_execution engine. What does that do and will that have any affect on some Ruby code I'm using to calculate the task\_id? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 21, 2020, 3:31pm UTC](https://discuss.elastic.co/t/aggregate-messages-in-sequence-order/220251/4 "2020-02-21T15:31:42Z")
The java\_execution engine was introduced a couple of years back to replace the ruby execution engine. There is a [bug](https://github.com/elastic/logstash/issues/10938) that causes it to re-order events. ---
**Author:** ![jfro](https://avatars.discourse-cdn.com/v4/letter/j/e95f7d/32.png) [@jfro](https://discuss.elastic.co/u/jfro)\ **Post date:** [February 26, 2020, 7:26pm UTC](https://discuss.elastic.co/t/aggregate-messages-in-sequence-order/220251/5 "2020-02-26T19:26:11Z")
Just wanted to follow up that we solved this by setting the listening node to 1 worker. This is in addition to setting the pipeline.workers to 1. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 25, 2020, 7:26pm UTC](https://discuss.elastic.co/t/aggregate-messages-in-sequence-order/220251/6 "2020-03-25T19:26:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.