Hello,
i have problems with input from a file.
Info: logstash 5.5, OS centos7
Problem: if i leave only one conf file in logstash/conf.d - it works,but if i put them together it's starting to fail. (conf files and part from logs- added. )
example, when parsing correct input to csv file it's add [tags ["_grokparsefailure"]] others fields - ok.(in kibana),
But if i insert bad data to file, i get error in /var/log/logstash/logstash-plain.log : [WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=> .... and then fail is starting, seems that logstash.filters.csv trying to parse logstash-plain.log warn log, and it get's warn that failed to parse, so it's closed circle.
HOW to make, that config 10-test.conf will parse only his files, and syslog config parse only his files?
i created first input(10-test.conf), from file (csv type) (short version of config) //everything works if there only one config//
input {
file {
path => "/var/log/test1/*.log"
start_position => end
sincedb_path => "/dev/null"
} }
filter { csv {
separator => ","
columns => ["TIMESTAMP","EVENT_TYPE","MSG_SIZE" .... and etc ]
convert => { "MSG_SIZE" => "integer" }
remove_field => [ "field_1", "field_2", and ect ]
}
// there some replaces x20
if [MSG_TYPE] == "1"{ mutate { replace => [ "MSG_TYPE", "newfield1" ] }}
}
output {
udp {
host => "192.168.100.152"
port => 12207
}
}
second config, from file (99-syslog.conf) ( /var/log/logstash/logstash-plain.log) // everything works if there only one config //
input {
file {
type => "syslog"
path => "/var/log/logstash/logstash-plain.log"
start_position => end
}}
filter {
grok {
match => { "message" => "[%{TIMESTAMP_ISO8601:timestamp}][%{LOGLEVEL:loglevel}%{SPACE}][%{DATA:logstash_filter}%{SPACE}] %{WORD:status }} } }
output {
udp {
host => "192.168.100.152"
port => 12207
} }