# Cluster eventually starts giving absurdly wrong counts on search

**URL:** <https://discuss.elastic.co/t/cluster-eventually-starts-giving-absurdly-wrong-counts-on-search/252096>\
**Category:** Elasticsearch\
**Created:** [October 14, 2020, 7:10pm UTC](https://discuss.elastic.co/t/cluster-eventually-starts-giving-absurdly-wrong-counts-on-search/252096 "2020-10-14T19:10:24Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![caarlos0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/caarlos0/32/77210_2.png) [@caarlos0](https://discuss.elastic.co/u/caarlos0)\
**Post date:** [October 14, 2020, 7:10pm UTC](https://discuss.elastic.co/t/cluster-eventually-starts-giving-absurdly-wrong-counts-on-search/252096/1 "2020-10-14T19:10:24Z")

</div>

We have a cluster running version 5.6.16. It has ~5.7k primary shards, ~2k indices and 28 nodes (3 masters, 3 coordinators and 22 data nodes):

```auto
{
  "cluster_name": "foo",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 28,
  "number_of_data_nodes": 22,
  "active_primary_shards": 5778,
  "active_shards": 11556,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

```

Eventually, whatever search we do in some indices return an insanely high doc count, even when no results are found.

For instance:

```auto
curl -s 'http://localhost:9200/*/_search?q=nope:thiswillneverexist&terminate_after=1' | jq -r '.'
{
  "took": 871,
  "timed_out": false,
  "terminated_early": false,
  "num_reduce_phases": 12,
  "_shards": {
    "total": 5778,
    "successful": 5778,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 9787770,
    "max_score": 2,
    "hits": []
  }
}

```

I couldn't find anything in the logs or any correlation with anything else, nor any other issues/foruns/etc (maybe I don't know what to search for exactly).

The only workaround we found was to restart all the data nodes ☹

Anyway, has everyone seen anything like this? Anything I could investigate?

Thanks!

---

<div class="post-metadata">

**Author:** ![caarlos0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/caarlos0/32/77210_2.png) [@caarlos0](https://discuss.elastic.co/u/caarlos0)\
**Post date:** [October 25, 2020, 1:20pm UTC](https://discuss.elastic.co/t/cluster-eventually-starts-giving-absurdly-wrong-counts-on-search/252096/2 "2020-10-25T13:20:27Z")

</div>

FWIW, this happens even when no results should be found, and the count, once the issue happens, is always the same...

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 25, 2020, 3:49pm UTC](https://discuss.elastic.co/t/cluster-eventually-starts-giving-absurdly-wrong-counts-on-search/252096/3 "2020-10-25T15:49:04Z")

</div>

That looks weird.

Unfortunately this version is not maintained anymore so even if it's a bug, it won't be fixed.

I wonder if after upgrading to 6.8 or better 7.9 you can still see this problem.

---

<div class="post-metadata">

**Author:** ![caarlos0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/caarlos0/32/77210_2.png) [@caarlos0](https://discuss.elastic.co/u/caarlos0)\
**Post date:** [October 28, 2020, 2:06am UTC](https://discuss.elastic.co/t/cluster-eventually-starts-giving-absurdly-wrong-counts-on-search/252096/4 "2020-10-28T02:06:05Z")

</div>

> Unfortunately this version is not maintained anymore so even if it's a bug, it won't be fixed.

yes, that was my fear...

> I wonder if after upgrading to 6.8 or better 7.9 you can still see this problem.

Not that easy to do in our case unfortunately...

* * *

Still hope I can find some workaround a bit better than restarting the whole cluster though 🙁

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 25, 2020, 2:06am UTC](https://discuss.elastic.co/t/cluster-eventually-starts-giving-absurdly-wrong-counts-on-search/252096/5 "2020-11-25T02:06:09Z")

</div>

Closed on November 25, 2020, 2:06am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.