I have been trial and error technique to solve problems with the logstash
I have been updating the config file input filter and output settings time to time.
so I want to index already present files but at first time it is taking but after I stops logstash and update config file it will taking only changes to files not the from begnnng
Hi,
Are you on linux or windows ?
both actually, I have to make confirm first on local windows and then on linux server
Ok so your problem seems to be coming from the sincedb_path directive which says that a file should be read at saved line or from the begining.
On linux it's something like "sincedb_path > /dev/null" ( read from the begining at each restart )
and windows "sincedb_path > NUL"
Try to look at the documentation good luck
# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
file{
path => "C:/ELK_Stack/logstash-7.4.0-1/bin/var/logs/*.log"
#start_position => "beginning"
codec => multiline{
#pattern => "^\s"
#what => "previous"
pattern => "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
negate => true
what => "previous"
}
}
}
filter{
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:time_stamp}\s%{WORD:log_level}\s%{JAVACLASS:class}\s(\[%{DATA:thread}\])\s+(?<msg>(.|\r|\n)*)"}
}
mutate{
gsub => ["time_stamp", " ","T"]
}
mutate{
gsub => ["time_stamp", ",","."]
}
mutate{
replace => {"time_stamp" => "%{time_stamp}Z"}
}
}
output {
stdout{
codec => rubydebug
}
elasticsearch {
hosts => ["http://localhost:9200"]
index => "localtest3"
}
}
this is my config file for windows
where should I put sincedb option
As @grumo35 said, it is explained in the documentation, if you want to read a file again you need to set the sincedb_path
to NUL
in windows and /dev/null
in linux.
This setting goes inside the file
input settings, you also needs to uncomment the start_position
setting.
So, try something like this:
input {
file {
path => "C:/ELK_Stack/logstash-7.4.0-1/bin/var/logs/*.log"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => "^[0-9]{4}-[0-9]{2}-[0-9]{2}"
negate => true
what => "previous"
}
}
}
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.