Updating config file

[mangeshs (Mangesh Shinde)]

I have been trial and error technique to solve problems with the logstash
I have been updating the config file input filter and output settings time to time.
so I want to index already present files but at first time it is taking but after I stops logstash and update config file it will taking only changes to files not the from begnnng

[grumo35]

Hi,

Are you on linux or windows ?

File input plugin | Logstash Reference [7.15] | Elastic

[mangeshs (Mangesh Shinde)]

both actually, I have to make confirm first on local windows and then on linux server

[grumo35]

Ok so your problem seems to be coming from the sincedb_path directive which says that a file should be read at saved line or from the begining.

On linux it's something like "sincedb_path > /dev/null" ( read from the begining at each restart )
and windows "sincedb_path > NUL"

Try to look at the documentation good luck

[mangeshs (Mangesh Shinde)]

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {

  file{
	path => "C:/ELK_Stack/logstash-7.4.0-1/bin/var/logs/*.log"
	#start_position => "beginning"
	codec => multiline{
			#pattern => "^\s"
			#what => "previous"
			pattern => "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
			negate => true
			what => "previous"
	}
  }
}
filter{
	grok {
		match => {"message" => "%{TIMESTAMP_ISO8601:time_stamp}\s%{WORD:log_level}\s%{JAVACLASS:class}\s(\[%{DATA:thread}\])\s+(?<msg>(.|\r|\n)*)"}
	}
	mutate{
		gsub => ["time_stamp", " ","T"]
	}
	mutate{
		gsub => ["time_stamp", ",","."]
	}
	mutate{
		replace => {"time_stamp" => "%{time_stamp}Z"}
	}
}
output {
	stdout{
		codec => rubydebug
	}
	elasticsearch {
		hosts => ["http://localhost:9200"]
		index => "localtest3"
  }
}

this is my config file for windows
where should I put sincedb option

[leandrojmp (Leandro Pereira)]

As @grumo35 said, it is explained in the documentation, if you want to read a file again you need to set the sincedb_path to NUL in windows and /dev/null in linux.

This setting goes inside the file input settings, you also needs to uncomment the start_position setting.

So, try something like this:

input {
    file {
        path => "C:/ELK_Stack/logstash-7.4.0-1/bin/var/logs/*.log"
        start_position => "beginning"
        sincedb_path => "NUL"
        codec => multiline {
            pattern => "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
            negate => true
            what => "previous"
        }
    }
}