# Logstash 2.1.0 with ElasticSearch 1.7.3 vs 2.0 vs 2.1 missing data

**URL:** <https://discuss.elastic.co/t/logstash-2-1-0-with-elasticsearch-1-7-3-vs-2-0-vs-2-1-missing-data/36539>\
**Category:** Logstash\
**Created:** [December 7, 2015, 2:40pm UTC](https://discuss.elastic.co/t/logstash-2-1-0-with-elasticsearch-1-7-3-vs-2-0-vs-2-1-missing-data/36539 "2015-12-07T14:40:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adam\_Wrobel](https://avatars.discourse-cdn.com/v4/letter/a/8baadc/32.png) [@Adam\_Wrobel](https://discuss.elastic.co/u/Adam_Wrobel)\
**Post date:** [December 7, 2015, 2:40pm UTC](https://discuss.elastic.co/t/logstash-2-1-0-with-elasticsearch-1-7-3-vs-2-0-vs-2-1-missing-data/36539/1 "2015-12-07T14:40:57Z")

</div>

Hi.  
I have ES cluster running on 1.7.3 storing logs parsed by logstash 2.1.  
I prepare new logstash template compatible with ES 2.x and run new separated ES cluster with version 2.0.1 and another one separated with 2.1.  
Logs are send to 3 clusters with this part of code:

> output {  
> elasticsearch {  
> hosts =\> "cluster17.es.service.consul"  
> template =\> "/etc/logstash/template\_api.json"  
> index =\> "logstash-%{[@context][\_index]}-%{+YYYY.MM.dd}"  
> template\_overwrite =\> true  
> flush\_size =\> 2000  
> retry\_max\_interval =\> 15  
> max\_retries =\> 6  
> }  
> elasticsearch {  
> hosts =\> "cluster20.es.service.consul"  
> template =\> "/etc/logstash/template\_api2.json"  
> index =\> "logstash-%{[@context][\_index]}-%{+YYYY.MM.dd}"  
> template\_overwrite =\> true  
> flush\_size =\> 2000  
> retry\_max\_interval =\> 15  
> max\_retries =\> 6  
> }  
> elasticsearch {  
> hosts =\> "cluster21.es.service.consul"  
> template =\> "/etc/logstash/template\_api2.json"  
> index =\> "logstash-%{[@context][\_index]}-%{+YYYY.MM.dd}"  
> template\_overwrite =\> true  
> flush\_size =\> 2000  
> retry\_max\_interval =\> 15  
> max\_retries =\> 6  
> }  
> }

And I had weird problem.  
In 1.7 cluster index from 1 day had:  
logstash-api-2015.12.06 items: 11,473,555 size: 5.3GB  
In 2.0.1 cluster:  
logstash-api-2015.12.06 items: 9,609,880 size: 4.7GB  
In 2.1 cluster:  
logstash-api-2015.12.06 items: 9,608,696 size: 4.6GB

Difference between ES 1.7 and 2.x is huge. And for each full daily indexes 2.x had 15-18% less data.

I tested ES 2.X on different hardware hosts/vms to exclude hardware problems. Also there was no errors in logs.  
I wrote script to compare indexes from 1.7 and 2.x and check what type of message is missing. But for each missing message I can POST it directly using curl to each cluster and everything saved without problems.  
ES cluster is green with no errors in logs. Logstash also have empty logs.  
I posted this problem on elasticsearch forum, but they suggested to post it here.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 8, 2015, 3:31am UTC](https://discuss.elastic.co/t/logstash-2-1-0-with-elasticsearch-1-7-3-vs-2-0-vs-2-1-missing-data/36539/2 "2015-12-08T03:31:52Z")

</div>

What does the rest of your LS config look like?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 8, 2015, 3:35am UTC](https://discuss.elastic.co/t/logstash-2-1-0-with-elasticsearch-1-7-3-vs-2-0-vs-2-1-missing-data/36539/3 "2015-12-08T03:35:39Z")

</div>

I can see you have also asked this here - [ElasticSearch 1.7.3 vs 2.0 vs 2.1 missing data](https://discuss.elastic.co/t/elasticsearch-1-7-3-vs-2-0-vs-2-1-missing-data/36502) - so I will close this one so we can continue the discussion here.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [December 8, 2015, 3:35am UTC](https://discuss.elastic.co/t/logstash-2-1-0-with-elasticsearch-1-7-3-vs-2-0-vs-2-1-missing-data/36539/4 "2015-12-08T03:35:44Z")

</div>