Hi,
I have two indexes, malware-*(contains a list of malware threats) and xxxx-main-controls, when the xxxx-2-3.conf (see below) runs it queries the malware index for a match to the urlpath field and if matched adds threat_category and url_link fields to the corresponding record in index xxxx-main-controls.
The first part works as expected, however it also dumps all the matched fields and data from the input JDBC to the malware-* index when this is supposed to be the lookup only index. I'm new to using the elasticsearch plugin filter so any help would be much appreciated.
xxxx-2-3.conf:
input {
jdbc connection multiple fields...........
filter {
if [type] == "xxxx-2-3" {
mutate {
convert => { "eventtime" => "string" }
add_field => {"xxxxtype" => "%{type}" }
}
date {
match => [ "eventtime", "ISO8601" ]
remove_field => [ "eventtime" ]
}
elasticsearch {
hosts => "http://xxx.xxx.xxx.xxx:9200"
index => "malware-*"
query => 'url:"%{[urlpath]}"'
fields => { "threat" => "threat_category" }
fields => { "urlhaus_link" => "url_link" }
}
}
}
output {
if [type] == "xxxx-2-3" {
elasticsearch {
hosts => "http://xxx.xxx.xxx.xxx:9200"
user => "ELASTIC_USER"
password => "ELASTIC_PWD"
index => "xxxx-main-controls-%{+YYYY.MM.dd}"
}
}
}