CCR - Remove follow_stats and License fetching problem

Elastic Stack Elasticsearch
1

Hi!
I'm playing around with the new cross cluster replication feature and ran into some issues. Firstly, it worked like a charm, data got replicated and everything was good. After that, I decided to put load on the leader index to see if the follower can catch up - spoiler, with default settings it couldn't.
Anyway, now I got the issue, that the follower index doesn't replicate any data anymore.
When i call _ccr/stats i get this message:

      {
        "leader_index": "cluster2-follows-cluster1",
        "timestamp": 1554360455200,
        "auto_follow_exception": {
          "type": "exception",
          "reason": "ElasticsearchStatusException[can not fetch remote cluster state as the license state of the remote cluster [elasticsearch] could not be determined]; nested: ResourceNotFoundException[license info is missing for cluster [elasticsearch]];",
          "caused_by": {
            "type": "status_exception",
            "reason": "can not fetch remote cluster state as the license state of the remote cluster [elasticsearch] could not be determined",
            "caused_by": {
              "type": "resource_not_found_exception",
              "reason": "license info is missing for cluster [elasticsearch]"
            }
          }
        }
      }
    ]

can this be the root cause why my data doesn't get replicated anymore?

Furthermore, when I call _ccr/stats i get the info about an Index I used to follow (visit). This index doesn't exist anymore but i the follower still wants to fetch data from it - resulting in a index not found exception. This was achieved with the auto-follow functionality. Is there a way to remove the "knowledge" of that index? It will never come back and I want to get rid of those exceptions.

The response looks something like this:

"follow_stats": {
    "indices": [
      {
        "index": "ccr-visit",
        "shards": [
          {
            "remote_cluster": "elasticsearch",
            "leader_index": "visit",
            "follower_index": "ccr-visit",
            "shard_id": 1,
            "leader_global_checkpoint": -1,
            ...
           }
      }
}

Thanks in advance!

And sorry, for the formatting,... I don't find formatting options anywhere :frowning:

2

can not fetch remote cluster state as the license state of the remote cluster [elasticsearch] could not be determined

If you're using a trial license, it may be that the trial license on the leader cluster expired.

Is there a way to remove the "knowledge" of that index?

yes, you can turn a follower index into a regular index (see Unfollow API | Elasticsearch Guide [8.11] | Elastic ). From that point on, you can also locally write into that index just as any other index.

3

Hi again :slight_smile:
I can tell for sure, that the trial is not expired - i started the trial on both clusters 3 days ago at the same time.
Regarding converting a follower index into a regular one. Both indices don't exist anymore and will never come back - was just for testing. Fun fact, I still get this exception when I create a leader index starting with visit-*. But, the leader index has a totally different name then the one mentioned in the exception.

[2019-04-05T09:55:12,683][WARN ][o.e.x.c.a.ShardFollowTasksExecutor] [node-2] [ccr-visit-...][2] background management of retention lease [elasticsearch/ccr-visit-.../H9bsXxlRTcSQzmqtVAUQYg-following-elasticsearch/visit-.../bBwfwk3JTIyPx0XwMvgkzA] failed while following
org.elasticsearch.index.IndexNotFoundException: no such index

On the setup - to the left, the leader index - to the right, the follower

image
image.png1071×357 14.2 KB

As you can see, the follower did not replicate any data, only created the index. The above warning occurs in the logs and i get the failed licence fetch message.
I delted the auto follow pattern and readded it, I restarted elasticsearch (yea i know) and it didn't help as well.
If you need any other logs or settings i used for this setup let me know.

4

Can you privately share the output of GET /_xpack/license of both clusters?

The license info is missing for cluster [elasticsearch] message is very surprising.

5

Hmm, ok, I called the license API and that's the response of cluster 1 ( node-1 = leader node):

{
  "license": {
    "status": "active",
    "uid": "e6ce6927-1247-4f31-a98e-...",
    "type": "trial",
    "issue_date": "2019-04-02T08:21:22.878Z",
    "issue_date_in_millis": 1554193282878,
    "expiry_date": "2019-05-02T08:21:22.878Z",
    "expiry_date_in_millis": 1556785282878,
    "max_nodes": 1000,
    "issued_to": "elasticsearch",
    "issuer": "elasticsearch",
    "start_date_in_millis": -1
  }
}

which looks good, but cluster 2 (node-2 = follower node) looks not so healthy:

{
  "error": {
    "root_cause": [
      {
        "type": "parse_exception",
        "reason": "failed to parse licenses expected field"
      }
    ],
    "type": "parse_exception",
    "reason": "failed to parse licenses expected field"
  },
  "status": 400
}
6

If this is a cluster for experimentation purposes only, perhaps remove / delete the data folder of node-2 and start afresh?

7

Yannick, we did it!
So, I deleted the (follower)node folder, which is ok because it's a test environment, had to re-activate the trial and it started working again.
Nevertheless, this will be a big nono in production. Is there maybe anything more I can share for you to maybe identify the root cause of this?

Thanks so far!

8

The symptoms and exception message suggest an expired trial license. If you still have the original data folder, you can privately share the data/nodes/0/_state/global-*.st file. Otherwise, if this occurs again, please run the start trial command to see if there is already a trial license in place, see https://www.elastic.co/guide/en/elasticsearch/reference/current/start-trial.html

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.