My input comes from a rest service that is digging in a database. Every time I request the service, I get events from the last 24 hours (for instance). That means, when I request the service every hour I will get a few same events as that got already with the last request.
I've learned that elesticsearch will override double events when the _id is the same. OK, then I try to set the document_id.
I send the data via a redis server. How can I set document_id in the config?
input {
stdin { }
exec {
codec => json
command => "C:\Data\RequestRest.exe"
interval => 20
}
}
filter {
split {
field => "results"
}
date {
match => ["[results][StartTime]","yyyy-MM-dd'T'HH:mm:ss.SSS"]
timezone => "UTC"
}
mutate {
add_field => {
"log_domain" => "cio"
"log_component" => "disflower_input_test"
}
}
}
output {
redis { host => "log4all-redis-sandbox.europe.intranet:6379"
data_type => "list"
key => "logstash"
}
stdout {
codec => rubydebug
}
}