Logstash input txt file received from filebeat is incorrect

Elastic Stack Logstash
1

Hello Community,

I am new to the community, and I need help.

I found that logstash is not receiving correctly from filebeat. Some of the lines are not in the correct order that they should be and other lines are missed, for example:

Source file (sent by filebeat):
8/4/2021,8:27 AM,10.64.58.119,LEVEL0,62791,77623
8/4/2021,8:27 AM,10.64.58.119A,LEVEL0,186300,183569
8/4/2021,8:27 AM,10.64.42.109,LEVEL 2,19328,17614
8/4/2021,8:27 AM,10.64.42.109A,LEVEL 2,235536,300359
8/4/2021,8:27 AM,10.64.42.109B,LEVEL 2,193146,243804
8/4/2021,8:31 AM,10.64.58.119,LEVEL0,62791,77623
8/4/2021,8:31 AM,10.64.58.119A,LEVEL0,186300,183569
8/4/2021,8:31 AM,10.64.42.109,LEVEL 2,19328,17614
8/4/2021,8:31 AM,10.64.42.109A,LEVEL 2,235537,300359
8/4/2021,8:31 AM,10.64.42.109B,LEVEL 2,193146,243804
8/4/2021,8:31 AM,10.64.58.119,LEVEL0,62791,77623
8/4/2021,8:31 AM,10.64.58.119A,LEVEL0,186300,183569
8/4/2021,8:31 AM,10.64.42.109,LEVEL 2,19328,17614
8/4/2021,8:31 AM,10.64.42.109A,LEVEL 2,235537,300359
8/4/2021,8:31 AM,10.64.42.109B,LEVEL 2,193147,243804

TXT file generated by logstash:
8/4/2021,8:27 AM,10.64.58.119,LEVEL0,62791,77623
8/4/2021,8:27 AM,10.64.58.119A,LEVEL0,186300,183569
8/4/2021,8:27 AM,10.64.42.109,LEVEL 2,19328,17614
8/4/2021,8:27 AM,10.64.42.109A,LEVEL 2,235536,300359
8/4/2021,8:27 AM,10.64.42.109B,LEVEL 2,193146,243804
8/4/2021,8:31 AM,10.64.58.119,LEVEL0,62791,77623
8/4/2021,8:31 AM,10.64.42.109,LEVEL 2,19328,17614
8/4/2021,8:31 AM,10.64.42.109B,LEVEL 2,193146,243804
8/4/2021,8:31 AM,10.64.58.119A,LEVEL0,186300,183569
8/4/2021,8:31 AM,10.64.42.109A,LEVEL 2,235537,300359
8/4/2021,8:31 AM,10.64.58.119A,LEVEL0,186300,183569
8/4/2021,8:31 AM,10.64.42.109A,LEVEL 2,235537,300359
8/4/2021,8:31 AM,10.64.58.119,LEVEL0,62791,77623
8/4/2021,8:31 AM,10.64.42.109,LEVEL 2,19328,17614
8/4/2021,8:31 AM,10.64.42.109B,LEVEL 2,193147,243804

I tried to re-sync by deleting the registry file of the filebeat, but nothing changed!

Thank you.

2

By default logstash does not preserve the order of events. If you want it to do so then set pipeline.workers to 1 and make sure pipeline.ordered has an appropriate value.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.