[illegal_argument_exception] unknown type for collapse field `cluster_uuid`, only keywords and numbers are accepted: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes

Elastic Stack Kibana
1

In Kibana when I click on Monitoring getting error
[illegal_argument_exception] unknown type for collapse field cluster_uuid, only keywords and numbers are accepted: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes.

In the Elasticsearch log file

[2020-03-22T10:40:25,595][DEBUG][o.e.a.s.TransportSearchAction] [ausplgicuesmon1_2dcmon] [.monitoring-es-7-mb-2020.03.22][0], node[zm8CYKFZQeuJSWmcoIfYNA], [R], s[STARTED], a[id=8Dm7sY0dRyqQrnX8Cho8pA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.monitoring-es-7-mb-2020.03.22], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":10000,"query":{"bool":{"filter":[{"term":{"type":{"value":"cluster_stats","boost":1.0}}},{"range":{"timestamp":{"from":1584888025379,"to":1584891625379,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"sort":[{"timestamp":{"order":"desc"}}],"collapse":{"field":"cluster_uuid"}}}]
org.elasticsearch.transport.RemoteTransportException: [ausflgicuesmon1_2dcmon][10.182.182.34:9301][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: unknown type for collapse field `cluster_uuid`, only keywords and numbers are accepted
        at org.elasticsearch.search.collapse.CollapseBuilder.build(CollapseBuilder.java:209) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:919) ~[elasticsearch-7.6.1.jar:7.6.1]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:591) ~[elasticsearch-7.6.1.jar:7.6.1]
2

What version of the stack are you using ? It looks like a mapping issue on the ES side.

Let's make sure you're indexing the right version of monitoring documents.

What do you see when you run:

GET _cat/indices/.monitoring-es-*

Try running this:

GET .monitoring-es-*/_mapping

The cluster_uuid mapping needs to be a keyword, and I'm guessing it's not for your setup:

        "cluster_uuid" : {
          "type" : "keyword"
        },

Is it safe to assume that all the existing monitoring data can be deleted without issue? Or do we need to preserve any of it? I'm asking because my recommendation moving forward will be to remove all .monitoring-es-* indices as a way to fix this.

Thanks
Rashmi

3

Hi Rashmi,

I am on latest version 7.6.1. Both the Production and Monitoring cluster are in same version 7.6.1

GET _cat/indices/.monitoring-es-*

green open .monitoring-es-7-mb-2020.03.23 1U3X7HffQYSsLnmwuvOgWA 1 1 108221 0 311.1mb 155.1mb
green open .monitoring-es-7-mb-2020.03.22 7d9ihmoHSDugbKwP09S6-Q 1 1 24977 0 70.1mb 34.9mb

GET .monitoring-es-*/_mapping

"cluster_stats": {
"properties": {
"cluster_uuid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}

4

@shaunak - can I seek ur inputs here?

Thanks
Rashmi

5

The mappings look wrong. Could you post the output of GET _cat/templates/*monitoring* ?v, run against your Monitoring ES cluster, please?

6

Hi Shaunak,

GET _cat/templates?v

name index_patterns order version
metricbeat-7.6.1 [metricbeat-7.6.1-] 1
.management-beats [.management-beats] 0 70000
.triggered_watches [.triggered_watches] 2147483647
.watches [.watches*] 2147483647
.watch-history-10 [.watcher-history-10*] 2147483647
.ml-anomalies- [.ml-anomalies-] 0 7060199
ilm-history [ilm-history-1] 2147483647
.ml-meta [.ml-meta] 0 7060199
.ml-inference-000001 [.ml-inference-000001] 0 7060199
.transform-internal-004 [.transform-internal-004] 0 7060199
.ml-notifications-000001 [.ml-notifications-000001] 0 7060199
.slm-history [.slm-history-1*] 2147483647
.transform-notifications-000002 [.transform-notifications-] 0 7060199
.ml-state [.ml-state] 0 7060199
.ml-config [.ml-config] 0 7060199
.logstash-management [.logstash] 0

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.