HI to all
I will try to explain this issue:
I have these lines in log
08:41:39.691 new resources port:6215 trunk:1
08:41:40.128 Release resources port:6215
The First line is index at 08:46, so late i think, but with the second line is worst, it is a delete for the doc and it takes sometimes hours to dissapear from elastic
My conf:
input {
file {
mode => "tail"
path => "/logs/*.log"
start_position => "beginning"
}
}
filter {
if [message] =~ "new resources" {
grok {
match=>["message","%{DATESTAMP:timestart}.*?\port:%{NUMBER:port}.*?\trunk:%{NUMBER:trunk}"]
}
date
{
locale => "es"
match => ["timestart","dd/MM/YY HH:mm:ss.SSS", "ISO8601"]
timezone => "Europe/Vienna"
target => "timestart"
}
mutate {
remove_field => ["@version"]
remove_field => ["host"]
remove_field => ["message"]
}
}
else if [message] =~ "Release resources" {
grok {
match=>["message","%{DATESTAMP:timeend}.*?\port:%{NUMBER:port}"]
}
mutate {
remove_field => ["@version"]
remove_field => ["host"]
remove_field => ["message"]
}
}else{
drop{}}
}
output {
stdout {}
if [trunk]
{
elasticsearch {
hosts => ["bbdd01.local:9200","bbdd02.local:9200","bbdd03.local:9200"]
index => "ipm_ory"
user => "logstash"
password => "pass"
document_id => "%{port}"
doc_as_upsert => true
action => index
retry_on_conflict => 5
}
}
if [timeend]
{
elasticsearch {
hosts => ["bbdd01.local:9200","bbdd02.local:9200","bbdd03.local:9200"]
index => "ipm_ory"
user => "logstash"
password => "pass"
document_id => "%{port}"
doc_as_upsert => true
action => delete
retry_on_conflict => 5
}
}
}
Maybe, is there some config in elastic to improve delete from logstash, or is there i way to improve my conf?
BR