# Error from \_cat APIs stemming from one bad data node

**URL:** <https://discuss.elastic.co/t/error-from-cat-apis-stemming-from-one-bad-data-node/210919>\
**Category:** Elasticsearch\
**Created:** [December 6, 2019, 4:43pm UTC](https://discuss.elastic.co/t/error-from-cat-apis-stemming-from-one-bad-data-node/210919 "2019-12-06T16:43:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nickcanz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickcanz/32/44873_2.png) [@nickcanz](https://discuss.elastic.co/u/nickcanz)\
**Post date:** [December 6, 2019, 4:43pm UTC](https://discuss.elastic.co/t/error-from-cat-apis-stemming-from-one-bad-data-node/210919/1 "2019-12-06T16:43:47Z")

</div>

Elasticsearch cluster running 7.3.1. I was adding/removing some nodes to the cluster, but no other config changes took place. The `_cat/nodes` API started returning an error from _all_ nodes. Nothing in any of the log files across the cluster:

```auto
$ curl localhost:9200/_cat/nodes
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -16b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -16b"},"status":400}

```

The `_cat/indices` API would just hang forever and not return results. Other `_cat` APIs and `_nodes` seemed to return fine.

I tracked it down to a single data node and this error:

```auto
$ curl http://localhost:9200/_nodes/_local/stats
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -16b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -16b","suppressed":[{"type":"illegal_state_exception","reason":"Failed to close the XContentBuilder","caused_by":{"type":"i_o_exception","reason":"Unclosed object or array found"}}]},"status":400}

```

After restarting that data node, the `_cat` APIs returned to normal.

1. Is there a way in the future I can better debug this type of error?
2. Is it expected that a problem on one node would break the `_cat` APIs for every node across the cluster?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 9, 2019, 2:12pm UTC](https://discuss.elastic.co/t/error-from-cat-apis-stemming-from-one-bad-data-node/210919/2 "2019-12-09T14:12:07Z")

</div>

if that occurs again, can you include the stack trace please. Just run ` curl http://localhost:9200/_nodes/_local/stats?error_trace=true` and share the output. Thanks!

---

<div class="post-metadata">

**Author:** ![nickcanz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickcanz/32/44873_2.png) [@nickcanz](https://discuss.elastic.co/u/nickcanz)\
**Post date:** [December 9, 2019, 2:37pm UTC](https://discuss.elastic.co/t/error-from-cat-apis-stemming-from-one-bad-data-node/210919/3 "2019-12-09T14:37:35Z")

</div>

Oh, I didn't know about that option, thanks!

---

<div class="post-metadata">

**Author:** ![loren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/loren/32/44942_2.png) [@loren](https://discuss.elastic.co/u/loren)\
**Post date:** [December 9, 2019, 11:24pm UTC](https://discuss.elastic.co/t/error-from-cat-apis-stemming-from-one-bad-data-node/210919/4 "2019-12-09T23:24:54Z")

</div>

I ran into this, too, a few weeks ago when ~20TB worth of shards were rebalancing after replacing some nodes on a 7.2 cluster. For me, it was a transient condition that fixed itself within the hour. I didn't know about that stack trace trick, either, but will keep it in mind if I run into this again.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 6, 2020, 11:25pm UTC](https://discuss.elastic.co/t/error-from-cat-apis-stemming-from-one-bad-data-node/210919/5 "2020-01-06T23:25:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.