Strange Marvel license issue

Elastic Stack Elasticsearch
1

I have an Elasticsearch 2.1.0 cluster that's been up and running with Marvel reporting to a separate monitoring cluster for around 5 months with no issues. But I'm now unable to access Marvel due to a strange license issue. Somehow the trial license was reactivated and is complaining that it's expired, even though I have a valid basic license installed. I can see that the marvel data is still being sent to and stored by the monitoring cluster, I'm just not able to access the Marvel UI.

Screenshot 2016-05-13 11.46.31.png789×334 23.7 KB

All of the nodes in my Elasticsearch cluster show a single installed basic license

$ http :9200/_license
HTTP/1.1 200 OK
Content-Length: 404
Content-Type: application/json; charset=UTF-8

{
    "license": {
        "expiry_date": "2017-01-04T23:59:59.999Z",
        "expiry_date_in_millis": 1483574399999,
        "issue_date": "2015-12-04T00:00:00.000Z",
        "issue_date_in_millis": 1449187200000,
        "issued_to": "--REDACTED--",
        "issuer": "Web Form",
        "max_nodes": 100,
        "status": "active",
        "type": "basic",
        "uid": "--REDACTED--"
    }
}

The only thing that changed before this occurred was that I removed some old .marvel-es-YYYY.MM.DD indexes from the monitoring cluster to free up space. I've done this before without any problems, so I'm doubtful that was the cause.

The only errors I'm seeing in any of the logs are from the monitoring cluster Elasticsearch logs when attempting to access the Marvel UI

[2016-05-13 19:02:48,115][DEBUG][action.search.type       ] [es-node-01] [.marvel-es-2016.05.01][0], node[s7vDPACDTm2bdfBmhWk13Q], [P], v[4], s[STARTED], a[id=w6K3YWkdRE2OsvzDA0jj6g]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7ba00ff3] lastShard [true]
RemoteTransportException[[es-node-01][10.1.2.66:9300][indices:data/read/search[phase/query]]]; nested: SearchParseException[failed to parse search source [{"size":0,"query":{"filtered":{"filter":{"term":{}}}},"aggs":{"indices":{"meta":{"cluster_uuid":"X0qHCkTuSVGs5SEoCCUz_w"},"terms":{"field":"shard.index","size":10000},"aggs":{"states":{"terms":{"field":"shard.state","size":10},"aggs":{"primary":{"terms":{"field":"shard.primary","size":10}}}}}},"nodes":{"meta":{"cluster_uuid":"X0qHCkTuSVGs5SEoCCUz_w"},"terms":{"field":"shard.node","size":10000},"aggs":{"index_count":{"cardinality":{"field":"shard.index"}}}}}}]]; nested: QueryParsingException[No value specified for term query];
Caused by: SearchParseException[failed to parse search source [{"size":0,"query":{"filtered":{"filter":{"term":{}}}},"aggs":{"indices":{"meta":{"cluster_uuid":"X0qHCkTuSVGs5SEoCCUz_w"},"terms":{"field":"shard.index","size":10000},"aggs":{"states":{"terms":{"field":"shard.state","size":10},"aggs":{"primary":{"terms":{"field":"shard.primary","size":10}}}}}},"nodes":{"meta":{"cluster_uuid":"X0qHCkTuSVGs5SEoCCUz_w"},"terms":{"field":"shard.node","size":10000},"aggs":{"index_count":{"cardinality":{"field":"shard.index"}}}}}}]]; nested: QueryParsingException[No value specified for term query];
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:848)
        ...
Caused by: [.marvel-es-2016.05.01] QueryParsingException[No value specified for term query]
        at org.elasticsearch.index.query.TermQueryParser.parse(TermQueryParser.java:108)
        ...

The index that it's complaining about exists and is full of documents. But the error indicates that Marvel is attempting to execute a bad query.

   $ http :9200/.marvel-es-2016.05.01/_count
    HTTP/1.1 200 OK
    Content-Length: 65
    Content-Type: application/json; charset=UTF-8

    {
        "_shards": {
            "failed": 0,
            "successful": 1,
            "total": 1
        },
        "count": 1493069
    }

Any help or advice would be much appreciated.
Thanks!
-Kevin

2

Hi Kevin,

The Basic license only allows a single cluster to be monitored, so what I suspect has happened is that the trial license ended up getting sent to Marvel while you were testing things out. As a result, it probably had a separate Cluster UUID, which is used to uniquely define the cluster (rather than using the cluster name for that purpose).

It looks like you're using Marvel 2.0 - 2.2 (they had the same index pattern), so you can probably get away with just cleaning the .marvel-es-data index:

DELETE host:9200/.marvel-es-data

The "Basic" cluster's nodes should then report within a minute. The elected master node will report the data necessary for this page to load.

3

That did the trick. Thanks for your help, Chris!
-Kevin