Logstash 2.1.0 with ElasticSearch 1.7.3 vs 2.0 vs 2.1 missing data


(Adam Wrobel) #1

Hi.
I have ES cluster running on 1.7.3 storing logs parsed by logstash 2.1.
I prepare new logstash template compatible with ES 2.x and run new separated ES cluster with version 2.0.1 and another one separated with 2.1.
Logs are send to 3 clusters with this part of code:

output {
elasticsearch {
hosts => "cluster17.es.service.consul"
template => "/etc/logstash/template_api.json"
index => "logstash-%{[@context][_index]}-%{+YYYY.MM.dd}"
template_overwrite => true
flush_size => 2000
retry_max_interval => 15
max_retries => 6
}
elasticsearch {
hosts => "cluster20.es.service.consul"
template => "/etc/logstash/template_api2.json"
index => "logstash-%{[@context][_index]}-%{+YYYY.MM.dd}"
template_overwrite => true
flush_size => 2000
retry_max_interval => 15
max_retries => 6
}
elasticsearch {
hosts => "cluster21.es.service.consul"
template => "/etc/logstash/template_api2.json"
index => "logstash-%{[@context][_index]}-%{+YYYY.MM.dd}"
template_overwrite => true
flush_size => 2000
retry_max_interval => 15
max_retries => 6
}
}

And I had weird problem.
In 1.7 cluster index from 1 day had:
logstash-api-2015.12.06 items: 11,473,555 size: 5.3GB
In 2.0.1 cluster:
logstash-api-2015.12.06 items: 9,609,880 size: 4.7GB
In 2.1 cluster:
logstash-api-2015.12.06 items: 9,608,696 size: 4.6GB

Difference between ES 1.7 and 2.x is huge. And for each full daily indexes 2.x had 15-18% less data.

I tested ES 2.X on different hardware hosts/vms to exclude hardware problems. Also there was no errors in logs.
I wrote script to compare indexes from 1.7 and 2.x and check what type of message is missing. But for each missing message I can POST it directly using curl to each cluster and everything saved without problems.
ES cluster is green with no errors in logs. Logstash also have empty logs.
I posted this problem on elasticsearch forum, but they suggested to post it here.


(Mark Walkom) #2

What does the rest of your LS config look like?


(Mark Walkom) #3

I can see you have also asked this here - ElasticSearch 1.7.3 vs 2.0 vs 2.1 missing data - so I will close this one so we can continue the discussion here.


(Mark Walkom) #4