Logstash is not pick up updated logs in newly created Log file with a same name after backup

[ldeepak (Deepak Lohar)]

Hi Team,

Please help me on below topic.

Logstash is not able to read updated logs after creating a new log file with the same and putting old log file as a backup file, this issue i had faced in windows.

please find below information for your reference.

1. Maximum size set for log file is 5MB.
2. Once 5MB gets full than new log file will create with same name and updated withe new logs in same file.
3. Newly created logfile is not picking up by logstash.
4. older file gets rename with current timestamp and saved as a backup.

please let me know in case of any extra information required for your analysis.

Thanks in advance.
Deepak Lohar
Deepaklohar275@gmail.com

[harshbajaj16 (Harsh Bajaj)]

Hi @ldeepak,

@ldeepak

please let me know in case of any extra information required for your analysis.

Could you please share your logstash input plugin configuration for more understanding?

Meanwhile, You need to set some parameter in your input plugin to read the file again from beginning. Please refer below post as they have explained in details how the input plugin works.

Importing the same file

Hi, I just have a conceptual question about how logstash import same files. Let's say I have done with the previous data ingesting. I terminated the process of logstash after that.And I do not change the data source file or corresponding config file. They are exactly same as before. If I restart the logstash with the same config setting, would the data be replaced by the new one(even if they are same) or logstash automatically recognizes they are same so it will not import any new data?

Regards,
Harsh Bajaj

[ldeepak (Deepak Lohar)]

Hi Harsh,

Thanks for your reply, here I am facing some different issue.

I have go through the post shared by you and below is my understanding.
In this They were discussing about file appending new log entries and we can keep track on current event ingested Logstash using SinceDB, as sincedb is already implemented in input section kindly check below.

Issue which i am facing that logstash is not able to read the logs as file is not getting append here but complete file is moved as a backup once 5MB gets filled and same file is start updating from 1st line at this time logstash is not able to pick the logs.

As requested please find input section below.

input {
** file {**
** path => ["LogFilePath"]**
** start_position => "beginning" **
** codec => multiline{**
** pattern => "%{NUMBER:Num} %{YEAR}-%{MONTHNUM}-%{MONTHDAY}"**
** negate => true**
** what => "previous"**
** } **
** add_field => "Extra Field added to the logs" **
** sincedb_path => "SinceDB File Path" **
** type => "Describing type of logs" **
** }**
}

Kindly let me know if above description is giving the clear understanding.

Thanks
Deepak