Logstash is not reading the log file(that's being updated automatically) continuously


(Amruth) #1

Hi, I have a log file that will be modified every 6 hours by a script. Problem is Logstash is not reading the file when the file is updated automatically. But when I update the file manually logstash is picking the data. I don't know the issue. Please help me on this issue. Here is my config file,

 input{
	file{
		path=>"C:/Path/Test.log"
		start_position=> "beginning"
		codec=> json
		sincedb_path => "NUL"
		ignore_older => 0
	}
}
output{
	stdout{
		codec=>rubydebug
		}
}

(Magnus Bäck) #2

Exactly how is the file modified by the script?


(Amruth) #3

It cleans the old data and writes new data every time and that is what I did manually and got it to work.


(Magnus Bäck) #4

That doesn't answer my question. Does the script just overwrite the old file? Is the file truncated? Before or after the file is truncated? These details matter.


(Amruth) #5

Hi Magnus, It just overwrites. File name remains same. What changes is just the data.


(Magnus Bäck) #6

Logstash doesn't discover such changes that happen behind its back. The reason it works for you is that your text editor (that I presume you used when you modified the file) actually creates a new file and renames it in place, replacing the original file. That is picked up by Logstash. The reason it's doing so is because it's typically a safer way of updating files that your script should consider employing.


(Amruth) #7

Magnus, I am little bit confused here. Are you saying that there is no way logstash can support this scenario? What would be the alternative? Thanks for your help.


(Magnus Bäck) #8

The file input rereads a file if

  • its inode number changes, or
  • if the file shrinks (indicating that it was rotated via copy/truncate).

The file input does not support other means of detecting changes in a file.


(Amruth) #9

if the file shrinks (indicating that it was rotated via copy/truncate) - Is this not my scenario? My script is truncating the data and then inserting new data.


(Magnus Bäck) #10

Sigh. But you just said that it just overwrites the file.

With debug logs enabled the file input says a few things about what it's doing. That might contain clues.


(Amruth) #11

Hi Magnus, sorry for the delay.

Here are my logs with debug logs enabled,

13:50:07.889 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:12.890 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:17.356 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:50:17.890 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:22.890 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:27.889 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:32.359 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:50:32.888 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:37.887 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:42.886 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:47.363 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:50:47.885 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:52.885 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:57.885 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:02.385 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:51:02.884 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:07.884 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:12.883 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:17.389 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:51:17.883 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:22.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:27.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:32.394 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:51:32.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:37.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:42.881 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:47.398 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:51:47.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:52.881 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:57.881 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:52:02.400 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:52:02.880 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:52:07.880 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:52:12.879 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:52:17.404 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]


(Amruth) #12

Hi Magnus, any update on this please... Please let me know if you need any other details.


(Patrick Stevens) #13

Hi Amruth, were you able to find a solution? I seem to be running into the same problem, and I can't find a clear solution anywhere.


(Amruth) #14

Hi Patrick, I don't have an answer yet!


(Patrick Stevens) #15

Ok, I'll keep you posted if I find anything. I'll give this a few more days, and then I may switch to a different log parser like Fluentd....


(Amruth) #16

Hi, can someone please help me on this issue?


(system) #17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.