Hi.
I have ES cluster running on 1.7.3 storing logs parsed by logstash 2.1.
I prepare new logstash template compatible with ES 2.x and run new separated ES cluster with version 2.0.1 and another one separated with 2.1.
Logs are send to 3 clusters with this part of code:
output {
elasticsearch {
hosts => "cluster17.es.service.consul"
template => "/etc/logstash/template_api.json"
index => "logstash-%{[@context][_index]}-%{+YYYY.MM.dd}"
template_overwrite => true
flush_size => 2000
retry_max_interval => 15
max_retries => 6
}
elasticsearch {
hosts => "cluster20.es.service.consul"
template => "/etc/logstash/template_api2.json"
index => "logstash-%{[@context][_index]}-%{+YYYY.MM.dd}"
template_overwrite => true
flush_size => 2000
retry_max_interval => 15
max_retries => 6
}
elasticsearch {
hosts => "cluster21.es.service.consul"
template => "/etc/logstash/template_api2.json"
index => "logstash-%{[@context][_index]}-%{+YYYY.MM.dd}"
template_overwrite => true
flush_size => 2000
retry_max_interval => 15
max_retries => 6
}
}
And I had weird problem.
In 1.7 cluster index from 1 day had:
logstash-api-2015.12.06 items: 11,473,555 size: 5.3GB
In 2.0.1 cluster:
logstash-api-2015.12.06 items: 9,609,880 size: 4.7GB
In 2.1 cluster:
logstash-api-2015.12.06 items: 9,608,696 size: 4.6GB
Difference between ES 1.7 and 2.x is huge. And for each full daily indexes 2.x had 15-18% less data.
I tested ES 2.X on different hardware hosts/vms to exclude hardware problems. Also there was no errors in logs.
I wrote script to compare indexes from 1.7 and 2.x and check what type of message is missing. But for each missing message I can POST it directly using curl to each cluster and everything saved without problems.
ES cluster is green with no errors in logs. Logstash also have empty logs.
I posted this problem on elasticsearch forum, but they suggested to post it here.