Logstash not parsing dynamic updates happening on the files present under the configured input path

[harish_92 (hari)]

Hi Guys,

I have configured a folder containing files which will be dynamically updated with some data which needs to be parsed by logstash and pushed into ES.

Here, whenever new data is written into any one of the input file, logstash is failing to detect and parse them(since db is also not updating). But if I do a restart of logstash it is able to detect the change and push the data into ES(since db is also updating).

It will be very helpful if someone can tell me how to make my logstash process the dynamic change happening in my configured file without requiring a restart of logstash.

My input config file looks like below:-

input { 
file {
path => "C:/Program Files (x86)/SapphireIMS/Plugins/LogAnalyzer/LogMon/Customlog/CustomlogFiles/*.csv"
start_position => "beginning"
sincedb_path => "C:/Program Files (x86)/SapphireIMS/Plugins/LogAnalyzer/logstash/bin/FileMon/.1_sincedb_customlog"
type => customlog
file_completed_action => ["log"]
file_completed_log_path => "C:/Program Files (x86)/SapphireIMS/Plugins/LogAnalyzer/LogMon/Customlog/CustomlogFiles/Archive.txt"
mode => ["read"]
}}

Note :: Tried to run the logstash with "--log.level trace" but couldnt find any error

Thanks in advance

[dadoonet (David Pilato)]

I moved your question to #elastic-stack:logstash

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

[harish_92 (hari)]

Thanks for guiding me on posting my question right. Hope my question is clear now.

[Badger]

@harish_92

Here, whenever new data is written into any one of the input file, logstash is failing to detect and parse them(since db is also not updating). But if I do a restart of logstash it is able to detect the change and push the data into ES(since db is also updating).

That sounds like a bug to me, but not the bug you think it is. In read mode it is assumed that the files do not change. If the files grow you should be using tail mode.

The file input reads the file and does not expect it to grow, so it does not go back to it. However, when it restarts, it loads the sincedb, checks the file, sees the file is longer than the length in the sincedb and processes the delta. I would argue that it should not do that.

[harish_92 (hari)]

Thank you so much. Now in tail mode, the dynamic updates are being picked up by the logstash without requiring to restart the logstash