the contents of the text file is just 4 lines for testing:
this is a test text
for logstash updation
adding another line
this is to check updates
should have 2 more doc counts
My logstash is running as a service and it takes 5 mins to update/push this data to the ES server (different servers).
Why is it not reflecting immediately for such small amount of data?
How do I improve this?
Hi Guys,
Additional information,
There seems to be no issues visible in the ES/Logstash logs.
In fact, no logs are getting generated when data update is happening.
Only when I restart a service, Logs get generated.
Kindly help me out if you have any idea about the above!
The file will be read and processed when Logstash starts up. Unless you add to it or change it it will not be reprocessed. If you are updating the file, how do you do this? It would be helpful if you described the exact steps you are taking and what you are seeing.
How are you updating the file? Logstash keeps track if how far it has read and keeps tailing the file. If you make changes to data already read it will not trigger a reprocessing unless you use an editor and it in effect shows up as a new file with the same name.
I am using a vi mytest.txt to that file and adding a few lines, that's it, I understand it wouldn't just upsert but entirely read the file again and process it. This is just a test and I don't have a primary key so I'm okay with it. What I do want to avoid is the 5 min delay, which I'm not sure of the reason. @Christian_Dahlqvist
Hi @Christian_Dahlqvist, apologies for the delayed response,
I see a term # pipeline: # batch: # size: 125 # delay: 5 # # Or as flat keys: # # pipeline.batch.size: 125 # pipeline.batch.delay: 5
in the logstash yml file, it is commented out, but however, could this be the default setting? does batch.delay: 5 mean 5 mins?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.