Elasticsearch with warm nodes setup, kibana not yet ready

Elastic Stack Elasticsearch
1

Hi

I setup the 7.8 version elastic search with 3 master, 3 client, 3 warm, 3 data nodes in amazon ec2 machines.

warm nodes:
bootstrap.memory_lock: true
node.name: ${HOSTNAME}

action.destructive_requires_name: true
indices.fielddata.cache.size: 1% # default is unbounded

Datanodes: 

bootstrap.memory_lock: true
node.name: ${HOSTNAME}

action.destructive_requires_name: true
indices.fielddata.cache.size: 1% # default is unbounded

masternode:
bootstrap.memory_lock: true
node.name: ${HOSTNAME}

action.destructive_requires_name: true
indices.fielddata.cache.size: 1% # default is unbounded
cluster.name: sfhlogging
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
path.data: /opt/elasticsearch/data
path.logs: /var/log/elasticsearch
xpack.security.enabled: false

network.host: _ec2:privateIpv4_,localhost
plugin.mandatory: discovery-ec2
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery:
    seed_providers: ec2
    ec2.groups: sg-018af72f8a8e37ec4
    ec2.host_type: private_ip
    ec2.tag.Cluster: dev-sfhlogging
    ec2.protocol: http # no need in HTTPS for internal AWS calls

    # manually set the endpoint because of auto-discovery issues
    # https://github.com/elastic/elasticsearch/issues/27464
    ec2.endpoint: ec2.us-east-1.amazonaws.com
node.master: true
node.data: false
node.ingest: false


clientnodes:
bootstrap.memory_lock: true
node.name: ${HOSTNAME}

action.destructive_requires_name: true
indices.fielddata.cache.size: 1% # default is unbounded
cluster.name: sfhlogging
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
path.data: /opt/elasticsearch/data
path.logs: /var/log/elasticsearch
xpack.security.enabled: false

network.host: _ec2:privateIpv4_,localhost
plugin.mandatory: discovery-ec2
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery:
    seed_providers: ec2
    ec2.groups: sg-018af72f8a8e37ec4
    ec2.host_type: private_ip
    ec2.tag.Cluster: dev-sfhlogging
    ec2.protocol: http # no need in HTTPS for internal AWS calls

    # manually set the endpoint because of auto-discovery issues
    # https://github.com/elastic/elasticsearch/issues/27464
    ec2.endpoint: ec2.us-east-1.amazonaws.com
node.master: false
node.data: false
node.ingest: false

I am able to access elasticsearch, grafana, cerebro. I am getting Kibana not yet ready message.

I mentioned the issue in below and got redirected here.

2

Is there anything in your Elasticsearch logs that mentions the .kibana index?
What is the output from the _cat/health?v API?

3 
epoch      timestamp cluster    status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1598455778 15:29:38  sfhlogging red             6         0      0   0    0    0        7             0                  -                  0.0%

journalctl -u elasticsearch
-- Logs begin at Fri 2020-07-24 00:11:43 UTC, end at Wed 2020-08-26 15:27:28 UTC. --
Aug 18 22:34:49 ip-10-128-44-63 systemd[1]: Starting Elasticsearch...
Aug 18 22:36:12 ip-10-128-44-63 systemd[1]: Started Elasticsearch.

Aug 24 19:41:48 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:41:48Z","tags":["info","plugins","watcher"],"pid":16069,"message":"Your basic license does not support watcher. Please upgrade your license."}
Aug 24 19:41:48 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:41:48Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":16069,"message":"Starting monitoring stats collection"}
Aug 24 19:41:48 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:41:48Z","tags":["info","savedobjects-service"],"pid":16069,"message":"Starting saved objects migrations"}
Aug 24 19:41:48 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:41:48Z","tags":["info","savedobjects-service"],"pid":16069,"message":"Creating index .kibana_task_manager_1."}
Aug 24 19:41:48 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:41:48Z","tags":["info","savedobjects-service"],"pid":16069,"message":"Creating index .kibana_1."}
Aug 24 19:42:18 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:42:18Z","tags":["warning","savedobjects-service"],"pid":16069,"message":"Unable to connect to Elasticsearch. Error: Request Timeout after 30000ms"}
Aug 24 19:42:20 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:42:20Z","tags":["warning","savedobjects-service"],"pid":16069,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_1/XSCzMemTSza0BDR2fdGNtA] already exists, with { index_uuid=\"XSCzMemTSza0BDR2fdGNtA\" & index=\".kibana_task_manager_1\" }"}
Aug 24 19:42:20 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:42:20Z","tags":["warning","savedobjects-service"],"pid":16069,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_1 and restarting Kibana."}
Aug 24 19:42:20 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:42:20Z","tags":["warning","savedobjects-service"],"pid":16069,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_1/k_twoA66Shm9QWXZEsDibg] already exists, with { index_uuid=\"k_twoA66Shm9QWXZEsDibg\" & index=\".kibana_1\" }"}
Aug 24 19:42:20 ip-10-128-44-63 kibana[16069]: {"type":"log","@timestamp":"2020-08-24T19:42:20Z","tags":["warning","savedobjects-service"],"pid":16069,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_1 and restarting Kibana."}

I stopped kibana and I deleted .kibana related indices. And restarted the kibana.
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.