Error since upgrade from 2.4.1 -> 5.6.4

Elastic Stack Elasticsearch
1

I recently upgraded from 2.4.1 to 5.6.4 (on my way to 6.0 eventually...) and I'm seeing the following error every 10 seconds in the elasticsearch log when I have any of the monitoring tabs open in kibana. If I switch to another "tab", they stop.

[2017-11-22T00:12:21,657][DEBUG][o.e.a.s.TransportSearchAction] [flash] [.marvel-es-1-2017.11.16][0], node[i6cjnXFET4C1onOiUVzHhg], [R], s[STARTED], a[id=s7ipKVaZT7SvjQhpRJ_Q_g]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.monitoring-es-2-*, .monitoring-es-6-*], indicesOptions=IndicesOptions[id=38, ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=10, batchedReduceSize=512, preFilterShardSize=128, source={
  "size" : 10000,
  "query" : {
    "bool" : {
      "filter" : [
        {
          "bool" : {
            "should" : [
              {
                "term" : {
                  "_type" : {
                    "value" : "cluster_stats",
                    "boost" : 1.0
                  }
                }
              },
              {
                "term" : {
                  "type" : {
                    "value" : "cluster_stats",
                    "boost" : 1.0
                  }
                }
              }
            ],
            "disable_coord" : false,
            "adjust_pure_negative" : true,
            "boost" : 1.0
          }
        },
        {
          "term" : {
            "cluster_uuid" : {
              "value" : "bphv2BscRQKlxA-t8z5szg",
              "boost" : 1.0
            }
          }
        },
        {
          "range" : {
            "timestamp" : {
              "from" : 1511305941595,
              "to" : 1511309541595,
              "include_lower" : true,
              "include_upper" : true,
              "format" : "epoch_millis",
              "boost" : 1.0
            }
          }
        }
      ],
      "disable_coord" : false,
      "adjust_pure_negative" : true,
      "boost" : 1.0
    }
  },
  "sort" : [
    {
      "timestamp" : {
        "order" : "desc"
      }
    }
  ],
  "collapse" : {
    "field" : "cluster_uuid"
  }
}}]
org.elasticsearch.transport.RemoteTransportException: [flash][172.16.0.100:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.SearchContextException: unknown type for collapse field `cluster_uuid`, only keywords and numbers are accepted
        at org.elasticsearch.search.collapse.CollapseBuilder.build(CollapseBuilder.java:240) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:743) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:485) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:461) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:343) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:340) ~[elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:258) [x-pack-5.6.4.jar:5.6.4]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:316) [x-pack-5.6.4.jar:5.6.4]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) [elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.4.jar:5.6.4]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.4.jar:5.6.4]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
2

I'm making a semi-educated guess, but maybe the .monitoring-es-2-* indices are causing issues. If you don't need them can you delete them?

3

They appear to all be aliases?

.monitoring-es-2-2017.11.19-alias .marvel-es-1-2017.11.19 - - -
.monitoring-es-2-2017.11.16-alias .marvel-es-1-2017.11.16 - - -
.monitoring-es-2-2017.11.18-alias .marvel-es-1-2017.11.18 - - -
.monitoring-es-2-2017.11.17-alias .marvel-es-1-2017.11.17 - - -
.monitoring-es-2-2017.11.20-alias .marvel-es-1-2017.11.20 - - -

Should I delete them and what they point to?

4

Unless you really want the data, yeah :slight_smile:

5

Thanks - I actually don't know what those indicies are... it's a monitoring (logs and host metrics) cluster. I'll give it a try.

6

This are from the Monitoring functionality in X-Pack.

7

I thot so. Deleting the aliases and what they pointed to did the trick. I didn't have x-pack installed prior to the upgrade - just sense.

I have to say the upgrade went way more smoothly than I thot it would :slight_smile:

Thanks for your help!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.