Inconsistent results in a histogram

Elastic Stack Elasticsearch
1

I'm wondering if anyone had seen something like this before: I have a 5 node cluster with a bunch of large date based indices (I.e. each daily index can have 50m docs). I have some kibana dashboards which visualize the data. Every so often one of the histogram graphs displays incorrect data - it creates a huge spike for one day and everything else is unreadable. The spike is incorrect (I.e. the elasticsearch data result is actually wrong). The weirder thing is that even on the same graph it is inconsistent - it looks OK at one point and then after a refresh it shows the bad data and then randomly switches back and forth. Is it possible that a replica or shard is corrupt? (The status of the cluster is green). Any thoughts on what may cause this or an approach for debugging it?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a8b80ad6-c839-4c5f-8f49-7dd04b1b91e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Just realized I forgot to mention: this is with version 1.4.4. Also I've
been able to localize it to a specific day's index, but I'm not sure what
the next steps should be to find the root cause and prevent/correct it.

On Monday, March 23, 2015 at 4:34:04 PM UTC-4, MC wrote:

I'm wondering if anyone had seen something like this before: I have a 5
node cluster with a bunch of large date based indices (I.e. each daily
index can have 50m docs). I have some kibana dashboards which visualize
the data. Every so often one of the histogram graphs displays incorrect
data - it creates a huge spike for one day and everything else is
unreadable. The spike is incorrect (I.e. the elasticsearch data result is
actually wrong). The weirder thing is that even on the same graph it is
inconsistent - it looks OK at one point and then after a refresh it shows
the bad data and then randomly switches back and forth. Is it possible
that a replica or shard is corrupt? (The status of the cluster is green).
Any thoughts on what may cause this or an approach for debugging it?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/31bc8d2c-174f-4a19-9351-e62723372529%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

You might try 1.5, which was released today. There was a bug fix for date histograms whereby improper DST handling could cause incorrect results.

On Mar 23, 2015, at 5:35 PM, MC mayer.crystal@gmail.com wrote:

Just realized I forgot to mention: this is with version 1.4.4. Also I've been able to localize it to a specific day's index, but I'm not sure what the next steps should be to find the root cause and prevent/correct it.

On Monday, March 23, 2015 at 4:34:04 PM UTC-4, MC wrote:
I'm wondering if anyone had seen something like this before: I have a 5 node cluster with a bunch of large date based indices (I.e. each daily index can have 50m docs). I have some kibana dashboards which visualize the data. Every so often one of the histogram graphs displays incorrect data - it creates a huge spike for one day and everything else is unreadable. The spike is incorrect (I.e. the elasticsearch data result is actually wrong). The weirder thing is that even on the same graph it is inconsistent - it looks OK at one point and then after a refresh it shows the bad data and then randomly switches back and forth. Is it possible that a replica or shard is corrupt? (The status of the cluster is green). Any thoughts on what may cause this or an approach for debugging it?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/31bc8d2c-174f-4a19-9351-e62723372529%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/31bc8d2c-174f-4a19-9351-e62723372529%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/B4303374-4DEE-4491-9316-0584EE42A61F%40elastic.co.
For more options, visit https://groups.google.com/d/optout.

4

I've found with K3 that if I'm updating filters but I don't wait for the
previous change to load completely that it seems that there are two
concurrent processes updating the histogram. Refreshing the page using the
in-page refresh, not the browser reload seems to resolve the display
problem for me.

Karl
On Mar 23, 2015 6:35 PM, "MC" mayer.crystal@gmail.com wrote:

Just realized I forgot to mention: this is with version 1.4.4. Also I've
been able to localize it to a specific day's index, but I'm not sure what
the next steps should be to find the root cause and prevent/correct it.

On Monday, March 23, 2015 at 4:34:04 PM UTC-4, MC wrote:

I'm wondering if anyone had seen something like this before: I have a 5
node cluster with a bunch of large date based indices (I.e. each daily
index can have 50m docs). I have some kibana dashboards which visualize
the data. Every so often one of the histogram graphs displays incorrect
data - it creates a huge spike for one day and everything else is
unreadable. The spike is incorrect (I.e. the elasticsearch data result is
actually wrong). The weirder thing is that even on the same graph it is
inconsistent - it looks OK at one point and then after a refresh it shows
the bad data and then randomly switches back and forth. Is it possible
that a replica or shard is corrupt? (The status of the cluster is green).
Any thoughts on what may cause this or an approach for debugging it?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/31bc8d2c-174f-4a19-9351-e62723372529%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/31bc8d2c-174f-4a19-9351-e62723372529%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BEXWsz-Gbq541Y04s1W9VCWPa6eGYYfZ4QbaA2_vsKdV%2B6Zag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.