Hi everyone,
I have 2 different application files to be monitored and on-boarded daily. I started off with creating 2 separate config files for the 2 applications, but have now moved to having 3 files - for input, filtering and output individually.
The filter and output conditions check for the 'type' field as created during input and apply plugins accordingly, however there is just one single field that is somehow propagating into the second index.
Below are my config files:
01-input.conf
input {
file {
path => "/path/to/f1"
type => "f1"
start_position => "beginning"
sincedb_path => "/path/to/f1/sincedb"
}
file {
path => "/path/to/f2"
type => "f2"
start_position => "beginning"
sincedb_path => "/path/to/f2/sincedb"
}
}
02-filter.conf:
filter {
if [type] == "f1" {
grok {
match => {
"message" => [
"pattern1",
"pattern2"
]
}
}
}
else if [type] == "f2" {
json {
source => "message"
}
}
}
03-outputs.conf:
output {
if [type] == 'f1' {
elasticsearch {
hosts => "http://localhost:9200"
index => "f1_ind"
document_type => "f1_type"
}
}
else if [type] == 'f2' {
elasticsearch {
hosts => "http://localhost:9200"
index => "f2_ind"
document_type => "f2_type"
}
}
}
The problem is:
Type F2 logs - are in JSON format and contain a key: "host", however F1 logs are plaintext and contain no such info.
In the f1_ind index - the documents contain the "host" key with values from F2 logs.
All other data within F2 is normal, the grok parsing on F1 is working fine as well. It is just the single 'host' field that is only present in F2 data that is showing up in both F1 and F2 indices.
I do not understand where I am going wrong and would strongly appreciate any help.