hi, guys
we use ELK
to analysis logs, recently we find that if we use the elasticsearch
output plugin, the total amount of logs is double, but if we use the amazon_es
output plugin, the total amount of logs is correct.
our configuration looks like:
input {
file {
path => "/data/web/test/*.info.log"
type => "info"
tags => ["old","new"]
}
}
filter {
grok {
patterns_dir => ["/etc/logstash/patterns", "/opt/logstash/patterns", "/opt/logstash/extra_patterns"]
match => { "message" => "%{TIMESTAMP_ISO8601:logtime}%{TK_TAB}%{GREEDYDATA:heading}%{TK_TAB}%{GREEDYDATA:data}"}
remove_field => ["message"]
}
date {
match => [ "logtime", "yyyy-MM-dd'T'HH:mm:ssZZ" ]
remove_field => ["logtime"]
}
json {
source => "data"
target => "@fields"
remove_field => ["data"]
}
metrics {
meter => "events"
add_tag => "metric"
}
}
output {
if "new" in [tags] {
amazon_es {
hosts => ["my_aws_es.amazonaws.com"]
region => "my_es_region"
aws_access_key_id => 'my_key_id'
aws_secret_access_key => 'my_key_secret'
index => "aws_es_test_%{+YYYY.MM.dd}"
document_type => "%{type}"
}
}
if "old" in [tags] {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "es_test_%{+YYYY.MM.dd}"
document_type => "%{type}"
}
}
}
On kibana
, we find that the total amount of logs which sent to the local elasticsearch
is double of the logs which sent to aws elasticsearch
Can anyone tell me why could this happen? any answers will be appreciated