How Logstash reads the same file name but with different content(keeps updating)?

Elastic Stack Logstash
1

Hi,

I am using logstash to read a log file. The file name reamins same but the content keeps changing for 1 hour(wipes off the previous logs and inserts new logs). But logstash is just reading last line of the file. Here is my config file,

        input{
        file {
        		path=>"C:logs/report.log"
        		start_position => "beginning"
        		type=>"ex"
        	}
        }
        output{
        if [type]  == "ex"{
        		stdout { 
        			codec => rubydebug 
        }
    }

Please help me on this issue. Thanks in advance.

2

It might be a sincedb issue? Logstash has logs to remember where it left off when parsing a certain file. Try going to your Logstash folder and clearing the sincedb folder (should be in "data"). Then do:
file { path=>"C:logs/report.log" start_position => "beginning" type=>"ex" sincedb_path => "NUL" }

Then try again. Hopefully that helps!

3

Hi, thanks for your reply. Will try that.

4

wipes off the previous logs and inserts new logs

Exactly how is this done? Is the original file overwritten or is the new data written to a new file that's renamed to replace the original file?

5

Hi Magnus, there is a cron job which replaces the content in the file every 1 hour, file name remains same. I've deleted sincedb folder and changed the config file but it still reads just the last line. Here is my config,

input{
file {
path=>"C:logs/report.log"
start_position => "beginning"
type=>"ex"
sincedb_path => "NUL"
}
}
output{
if [type] == "ex"{
stdout {
codec => rubydebug
}
}

6

Really, it's reading only the last line of the file even with sincedb_path => "NUL"?

7

I've seen some instances where it only reads the last event. But now it's not reading at all. When I manually delete the content in file and add some content, logstash is reading the content but it's not doing automatically. Also I am running logstash.conf from a different path(not from logstash bin), will this be causing any issue?

8

I've seen some instances where it only reads the last event. But now it's not reading at all. When I manually delete the content in file and add some content, logstash is reading the content but it's not doing automatically.

Bumping up the log level to debug and looking for everything having to do with the reading of the file (including sincedb) might give useful clues.

Also I am running logstash.conf from a different path(not from logstash bin), will this be causing any issue?

No.

9

Magnus, Can you please be more specific? i don't understand "Bumping up the log level to debug". Thank you.

10

Increase Logstash's log level, e.g. by starting it with --log.level debug.

11

I've started it with log level, but can you please say what extra information I can get from that?

12

Clues about what's going in inside Logstash, like what sincedb file (if any) it's reading, the current read position in a file, and so on.. I don't know what's wrong with your setup so I can't tell you exactly what to look for.

13

But I don't see any extra information like current read position, sincedb file.

14

Yes, but maybe someone else can spot the clues if you post the log to e.g. pastebin.

15

Magnus, I appreciate your patience. Please look at the logs below and let me know if those make sense,

[2017-06-27T14:46:22,523][DEBUG][logstash.inputs.file ] each: file grew: C:logs/another_report.log: old size 5924, new size 5931
[2017-06-27T14:46:23,517][DEBUG][logstash.pipeline ] Pushing flush onto p
ipeline
[2017-06-27T14:46:23,525][DEBUG][logstash.inputs.file ] each: file grew: C:logs/another_report.log: old size 5924, new size 5931
[2017-06-27T14:46:24,526][DEBUG][logstash.inputs.file ] each: file grew: C:logs/another_report.log: old size 5924, new size 5931
[2017-06-27T14:46:24,527][DEBUG][logstash.agent ] Reading config file
{:config_file=>"C:logs/log.conf"}
[2017-06-27T14:46:24,528][DEBUG][logstash.agent ] no configuration cha
nge for pipeline {:pipeline=>"main", :config=>"input{\n\tfile {\n\t\tpath=>"C:/

16

Let's see all the logs.

17

It's running continuously, they are pretty much the same logs that are being repeated. When you see the above logs, each:file grew...is another file in config, not the actual file I am looking for. (In the config I have multiple files)

18

Hi Magnus, Can you please help on this?

19

I asked for complete logs and you haven't provided them.

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.