Hi, I have a log file that will be modified every 6 hours by a script. Problem is Logstash is not reading the file when the file is updated automatically. But when I update the file manually logstash is picking the data. I don't know the issue. Please help me on this issue. Here is my config file,
input{
file{
path=>"C:/Path/Test.log"
start_position=> "beginning"
codec=> json
sincedb_path => "NUL"
ignore_older => 0
}
}
output{
stdout{
codec=>rubydebug
}
}Exactly how is the file modified by the script?
It cleans the old data and writes new data every time and that is what I did manually and got it to work.
That doesn't answer my question. Does the script just overwrite the old file? Is the file truncated? Before or after the file is truncated? These details matter.
Hi Magnus, It just overwrites. File name remains same. What changes is just the data.
Logstash doesn't discover such changes that happen behind its back. The reason it works for you is that your text editor (that I presume you used when you modified the file) actually creates a new file and renames it in place, replacing the original file. That is picked up by Logstash. The reason it's doing so is because it's typically a safer way of updating files that your script should consider employing.
Magnus, I am little bit confused here. Are you saying that there is no way logstash can support this scenario? What would be the alternative? Thanks for your help.
The file input rereads a file if
The file input does not support other means of detecting changes in a file.
if the file shrinks (indicating that it was rotated via copy/truncate) - Is this not my scenario? My script is truncating the data and then inserting new data.
Sigh. But you just said that it just overwrites the file.
With debug logs enabled the file input says a few things about what it's doing. That might contain clues.
Hi Magnus, sorry for the delay.
Here are my logs with debug logs enabled,
13:50:07.889 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:12.890 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:17.356 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:50:17.890 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:22.890 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:27.889 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:32.359 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:50:32.888 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:37.887 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:42.886 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:47.363 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:50:47.885 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:52.885 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:50:57.885 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:02.385 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:51:02.884 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:07.884 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:12.883 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:17.389 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:51:17.883 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:22.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:27.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:32.394 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:51:32.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:37.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:42.881 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:47.398 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:51:47.882 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:52.881 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:51:57.881 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:52:02.400 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
13:52:02.880 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:52:07.880 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:52:12.879 [Ruby-0-Thread-10: C:/logstash-5.3.1/logstash-5.3.1/logstash-core/lib/logstash/pipeline.rb:514] DEBUG logstash.pipeline - Pushing flush onto pipeline
13:52:17.404 [[main]<file] DEBUG logstash.inputs.file - _globbed_files: C:/Path/test.log: glob is: ["C:/Path/test.log"]
Hi Magnus, any update on this please... Please let me know if you need any other details.
Hi Amruth, were you able to find a solution? I seem to be running into the same problem, and I can't find a clear solution anywhere.
Hi Patrick, I don't have an answer yet!
Ok, I'll keep you posted if I find anything. I'll give this a few more days, and then I may switch to a different log parser like Fluentd....
Hi, can someone please help me on this issue?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.