# Incorrect Aggregations returned from ES

**URL:** <https://discuss.elastic.co/t/incorrect-aggregations-returned-from-es/22908>\
**Category:** Elasticsearch\
**Created:** [March 25, 2015, 10:57pm UTC](https://discuss.elastic.co/t/incorrect-aggregations-returned-from-es/22908 "2015-03-25T22:57:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![MC1](https://avatars.discourse-cdn.com/v4/letter/m/b3f665/32.png) [@MC1](https://discuss.elastic.co/u/MC1)\
**Post date:** [March 25, 2015, 10:57pm UTC](https://discuss.elastic.co/t/incorrect-aggregations-returned-from-es/22908/1 "2015-03-25T22:57:15Z")

</div>

I am seeing some erroneous behavior in my ES cluster when performing  
aggregations. Originally, I thought this was specific to a histogram as  
that is where the error first appeared (in a K3 graph - see my post  
[https://groups.google.com/forum/#!topic/elasticsearch/iY-lKjtW7PM](https://groups.google.com/forum/#!topic/elasticsearch/iY-lKjtW7PM) for  
reference) but I have been able to re-create the exception with a simple  
max aggregation. The details are as follows:

ES Version: 1.4.4  
Topology: 5 nodes, 5 shards per index, 2 replicas  
OS: Redhat Linux

To create the issue I execute the following query against the cluster:

{  
"query": {  
"term": {  
"metric": "used"  
}  
},  
"aggs": {  
"max\_val": {  
"max": {  
"field": "metric\_value"  
}  
}  
}  
}

Upon executing this query multiple times, I get different responses. One  
time I get the expected result:  
...  
"took": 13,  
"timed\_out": false,  
"\_shards": {  
"total": 5,  
"successful": 5,  
"failed": 0  
},  
"hits": {  
"total": 11712,  
"max\_score": 9.361205,  
...  
"aggregations": { "max\_val": { "value": 18096380}}

whereas on another request with the same query I get the following bad  
response:

"took": 8,  
"timed\_out": false,  
"\_shards": {  
"total": 5,  
"successful": 5,  
"failed": 0  
},  
"hits": {  
"total": 11712,  
"max\_score": 9.361205,  
...  
"aggregations": { "max\_val": { "value": 4697741490703565000}}

Some possibly relevant observations:

1. In my first set of tests, I was consistently getting the correct  
results for the first 2 requests and the bad result on the 3rd request  
(with no one else executing this query at that point in time)
2. Flushing the cache did not correct the issue
3. I reduced the number of replicas to 0 and was consistently getting the  
same result (which happened to be the correct one)
4. After increasing the replica count back to 2 and waiting until ES  
reported that the replication was complete, I tried the same experiment.  
This time, the 1st request retrieved the correct result and the next 2  
requests retrieved incorrect results. In this case the incorrect results  
were not the same but were both huge and of the same order of magnitude.

Other info:

- The size of the index was about 3.3Gb with ~ 50M documents in it
- This is one of many date based indices (i.e. similar to the logstash  
index setup), but the only one in this installation that exhibited the  
issue. I believe we saw something similar in a UAT environment as well  
where 1 or 2 of the indices acted in this weird manner
- ES reported the entire cluster as green

It seems that some shard(s)/replica(s) were being corrupted on the  
replication and we were being routed to that one every 3rd hit. (Is this  
somehow correlated to the number of replicas?)

So, my questions are:

1. Has anyone seen this type of behavior before?
2. Can it somehow be data dependent?
3. Is there any way to figure out what happened/what is happening?
4. Why does ES report the cluster state as green?
5. How can I debug this?
6. How can I prevent/correct this?

Any and all help/pointers would be greatly appreciated.

Thanks in advance,  
MC

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2461a3f0-aee4-45f7-9210-3ef3524b12c5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2461a3f0-aee4-45f7-9210-3ef3524b12c5%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![thanodnl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thanodnl/32/44907_2.png) [@thanodnl](https://discuss.elastic.co/u/thanodnl)\
**Post date:** [April 14, 2015, 1:55pm UTC](https://discuss.elastic.co/t/incorrect-aggregations-returned-from-es/22908/2 "2015-04-14T13:55:43Z")

</div>

Hi,

To me this sounds a lot like an issue that was happening to me a week  
before the release of 1.0.0. This issue was related to internal memory  
reuse within Elasticsearch before the result was read out. The issue is  
documented here: [Aggregations return different counts when invoked twice in a row · Issue #5021 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/5021)

What I did back then was create a reproducible test that showed the issue.

I doubt it has to do with your replica's being inconsistent. Especially  
since you turned off replicating and then turned it back on, this copies  
the files you have in your primary to the secondaries.

-- Nils  
Here is the test I created in the past:

> <https://gist.github.com/thanodnl/8803745>
>
> There are more than three files. show original

On Wednesday, March 25, 2015 at 11:57:15 PM UTC+1, MC wrote:

> I am seeing some erroneous behavior in my ES cluster when performing  
> aggregations. Originally, I thought this was specific to a histogram as  
> that is where the error first appeared (in a K3 graph - see my post  
> [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/elasticsearch/iY-lKjtW7PM) for  
> reference) but I have been able to re-create the exception with a simple  
> max aggregation. The details are as follows:
> 
> ES Version: 1.4.4  
> Topology: 5 nodes, 5 shards per index, 2 replicas  
> OS: Redhat Linux
> 
> To create the issue I execute the following query against the cluster:
> 
> {  
> "query": {  
> "term": {  
> "metric": "used"  
> }  
> },  
> "aggs": {  
> "max\_val": {  
> "max": {  
> "field": "metric\_value"  
> }  
> }  
> }  
> }
> 
> Upon executing this query multiple times, I get different responses. One  
> time I get the expected result:  
> ...  
> "took": 13,  
> "timed\_out": false,  
> "\_shards": {  
> "total": 5,  
> "successful": 5,  
> "failed": 0  
> },  
> "hits": {  
> "total": 11712,  
> "max\_score": 9.361205,  
> ...  
> "aggregations": { "max\_val": { "value": 18096380}}
> 
> whereas on another request with the same query I get the following bad  
> response:
> 
> "took": 8,  
> "timed\_out": false,  
> "\_shards": {  
> "total": 5,  
> "successful": 5,  
> "failed": 0  
> },  
> "hits": {  
> "total": 11712,  
> "max\_score": 9.361205,  
> ...  
> "aggregations": { "max\_val": { "value": 4697741490703565000}}
> 
> Some possibly relevant observations:
> 
> 1. In my first set of tests, I was consistently getting the correct  
> results for the first 2 requests and the bad result on the 3rd request  
> (with no one else executing this query at that point in time)
> 2. Flushing the cache did not correct the issue
> 3. I reduced the number of replicas to 0 and was consistently getting the  
> same result (which happened to be the correct one)
> 4. After increasing the replica count back to 2 and waiting until ES  
> reported that the replication was complete, I tried the same experiment.  
> This time, the 1st request retrieved the correct result and the next 2  
> requests retrieved incorrect results. In this case the incorrect results  
> were not the same but were both huge and of the same order of magnitude.
> 
> Other info:
> 
> - The size of the index was about 3.3Gb with ~ 50M documents in it
> - This is one of many date based indices (i.e. similar to the logstash  
> index setup), but the only one in this installation that exhibited the  
> issue. I believe we saw something similar in a UAT environment as well  
> where 1 or 2 of the indices acted in this weird manner
> - ES reported the entire cluster as green
> 
> It seems that some shard(s)/replica(s) were being corrupted on the  
> replication and we were being routed to that one every 3rd hit. (Is this  
> somehow correlated to the number of replicas?)
> 
> So, my questions are:
> 
> 1. Has anyone seen this type of behavior before?
> 2. Can it somehow be data dependent?
> 3. Is there any way to figure out what happened/what is happening?
> 4. Why does ES report the cluster state as green?
> 5. How can I debug this?
> 6. How can I prevent/correct this?
> 
> Any and all help/pointers would be greatly appreciated.
> 
> Thanks in advance,  
> MC

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/14c3bff7-b17b-4fa8-938f-cf8e13c80a29%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/14c3bff7-b17b-4fa8-938f-cf8e13c80a29%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [April 14, 2015, 2:05pm UTC](https://discuss.elastic.co/t/incorrect-aggregations-returned-from-es/22908/3 "2015-04-14T14:05:27Z")

</div>

Nils: It looks different from your issue since document counts are correct  
here?

MC: I think it is due to  
[Mapping updates should be synchronous · Issue #8688 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/8688). Has your `max_val`  
field been dynamically mapped? The only way to prevent this issue is to map  
fields explicitely instead of relying on dynamic mappings.

On Tue, Apr 14, 2015 at 3:55 PM, Nils Dijk [me@thanod.nl](mailto:me@thanod.nl) wrote:

> Hi,
> 
> To me this sounds a lot like an issue that was happening to me a week  
> before the release of 1.0.0. This issue was related to internal memory  
> reuse within Elasticsearch before the result was read out. The issue is  
> documented here: [Aggregations return different counts when invoked twice in a row · Issue #5021 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/5021)
> 
> What I did back then was create a reproducible test that showed the issue.
> 
> I doubt it has to do with your replica's being inconsistent. Especially  
> since you turned off replicating and then turned it back on, this copies  
> the files you have in your primary to the secondaries.
> 
> -- Nils  
> Here is the test I created in the past:  
> [https://gist.github.com/thanodnl/8803745](https://gist.github.com/thanodnl/8803745)
> 
> On Wednesday, March 25, 2015 at 11:57:15 PM UTC+1, MC wrote:
> 
> > I am seeing some erroneous behavior in my ES cluster when performing  
> > aggregations. Originally, I thought this was specific to a histogram as  
> > that is where the error first appeared (in a K3 graph - see my post  
> > [Redirecting to Google Groups](https://groups.google.com/forum/#!topic/elasticsearch/iY-lKjtW7PM) for  
> > reference) but I have been able to re-create the exception with a simple  
> > max aggregation. The details are as follows:
> > 
> > ES Version: 1.4.4  
> > Topology: 5 nodes, 5 shards per index, 2 replicas  
> > OS: Redhat Linux
> > 
> > To create the issue I execute the following query against the cluster:
> > 
> > {  
> > "query": {  
> > "term": {  
> > "metric": "used"  
> > }  
> > },  
> > "aggs": {  
> > "max\_val": {  
> > "max": {  
> > "field": "metric\_value"  
> > }  
> > }  
> > }  
> > }
> > 
> > Upon executing this query multiple times, I get different responses. One  
> > time I get the expected result:  
> > ...  
> > "took": 13,  
> > "timed\_out": false,  
> > "\_shards": {  
> > "total": 5,  
> > "successful": 5,  
> > "failed": 0  
> > },  
> > "hits": {  
> > "total": 11712,  
> > "max\_score": 9.361205,  
> > ...  
> > "aggregations": { "max\_val": { "value": 18096380}}
> > 
> > whereas on another request with the same query I get the following bad  
> > response:
> > 
> > "took": 8,  
> > "timed\_out": false,  
> > "\_shards": {  
> > "total": 5,  
> > "successful": 5,  
> > "failed": 0  
> > },  
> > "hits": {  
> > "total": 11712,  
> > "max\_score": 9.361205,  
> > ...  
> > "aggregations": { "max\_val": { "value": 4697741490703565000}}
> > 
> > Some possibly relevant observations:
> > 
> > 1. In my first set of tests, I was consistently getting the correct  
> > results for the first 2 requests and the bad result on the 3rd request  
> > (with no one else executing this query at that point in time)
> > 2. Flushing the cache did not correct the issue
> > 3. I reduced the number of replicas to 0 and was consistently getting  
> > the same result (which happened to be the correct one)
> > 4. After increasing the replica count back to 2 and waiting until ES  
> > reported that the replication was complete, I tried the same experiment.  
> > This time, the 1st request retrieved the correct result and the next 2  
> > requests retrieved incorrect results. In this case the incorrect results  
> > were not the same but were both huge and of the same order of magnitude.
> > 
> > Other info:
> > 
> > - The size of the index was about 3.3Gb with ~ 50M documents in it
> > - This is one of many date based indices (i.e. similar to the logstash  
> > index setup), but the only one in this installation that exhibited the  
> > issue. I believe we saw something similar in a UAT environment as well  
> > where 1 or 2 of the indices acted in this weird manner
> > - ES reported the entire cluster as green
> > 
> > It seems that some shard(s)/replica(s) were being corrupted on the  
> > replication and we were being routed to that one every 3rd hit. (Is this  
> > somehow correlated to the number of replicas?)
> > 
> > So, my questions are:
> > 
> > 1. Has anyone seen this type of behavior before?
> > 2. Can it somehow be data dependent?
> > 3. Is there any way to figure out what happened/what is happening?
> > 4. Why does ES report the cluster state as green?
> > 5. How can I debug this?
> > 6. How can I prevent/correct this?
> > 
> > Any and all help/pointers would be greatly appreciated.
> > 
> > Thanks in advance,  
> > MC
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/14c3bff7-b17b-4fa8-938f-cf8e13c80a29%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/14c3bff7-b17b-4fa8-938f-cf8e13c80a29%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/14c3bff7-b17b-4fa8-938f-cf8e13c80a29%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/14c3bff7-b17b-4fa8-938f-cf8e13c80a29%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Adrien

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhroYFe4uk%2BmeE0bbz86S73vRAZZ%2Bk462W%3DtN4t%2BQv5Bg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhroYFe4uk%2BmeE0bbz86S73vRAZZ%2Bk462W%3DtN4t%2BQv5Bg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:19am UTC](https://discuss.elastic.co/t/incorrect-aggregations-returned-from-es/22908/4 "2017-07-06T00:19:54Z")

</div>