Kibana NoShardsAvailable Exception

Elastic Stack Kibana
1

Hi Team,

I have been using Kibana to visualize and the last few days it is throwing error :NoShardsAvailableException, please find the snap below.

image
image865×688 45 KB

Note : Ignore the index name

Please note that I ran few queries to understand the unassigned shards and below is the output

curl -XGET  localhost:9200/_cat/shards?v | grep UNASSIGNED
0index_name-repolib-push-2020.10.22                                              0     r      UNASSIGNED
index_name-repolib-push-2020.10.22                                              0     p      UNASSIGNED
index_name-repoexec-team_add-2020.10.20                                         0     p      UNASSIGNED
index_name-repoexec-team_add-2020.10.20                                         0     r      UNASSIGNED
.kibana-event-log-7.9.1-000001                                           0     p      UNASSIGNED
.kibana-event-log-7.9.1-000001                                           0     r      UNASSIGNED
index_name-repo4-status-2020.10.22                                               0     p      UNASSIGNED
index_name-repo4-status-2020.10.22                                               0     r      UNASSIGNED
index_name-repo1-pull_request_review-2020.10.23                                   0     r      UNASSIGNED
index_name-repo1-pull_request_review-2020.10.23                                   0     p      UNASSIGNED
index_name-repo4-status-2020.10.30                                               0     r      UNASSIGNED
index_name-repo4-status-2020.10.30                                               0     p      UNASSIGNED
index_name-repo4-member-2020.10.21                                               0     r      UNASSIGNED
index_name-repo4-member-2020.10.21                                               0     p      UNASSIGNED
index_name-repo1-pull_request_review-2020.10.28                                   0     p      UNASSIGNED
index_name-repo1-pull_request_review-2020.10.28                                   0     r      UNASSIGNED
index_name-repo4-pull_request_review_comment-2020.10.25                          0     r      UNASSIGNED
index_name-pylibs-push                                                          0     p      UNASSIGNED
index_name-pylibs-push                                                          0     r      UNASSIGNED
index_name-repo2-push-2020.10.27                                            0     r      UNASSIGNED
index_name-repo2-push-2020.10.27                                            0     p      UNASSIGNED
index_name-repo4-status-2020.10.21                                               0     r      UNASSIGNED
index_name-repo4-status-2020.10.21                                               0     p      UNASSIGNED
index_name-repo4-pull_request_review-2020.10.20                                  0     r      UNASSIGNED
index_name-repo4-pull_request_review-2020.10.20                                  0     p      UNASSIGNED
index_name-repo4-status-2020.10.25                                               0     p      UNASSIGNED
index_name-repo4-status-2020.10.25                                               0     r      UNASSIGNED
index_name-repoexec-pull_request                                                0     r      UNASSIGNED
index_name-repoexec-pull_request                                                0     p      UNASSIGNED
index_name-repo4-pull_request_review_comment-2020.10.20                          0     p      UNASSIGNED
index_name-repo4-pull_request_review_comment-2020.10.20                          0     r      UNASSIGNED
index_name-repo1-push-2020.10.24                                                  0     p      UNASSIGNED
index_name-repo1-push-2020.10.24                                                  0     r      UNASSIGNED
index_name-repo1-pull_request_review_comment-2020.10.24                           0     r      UNASSIGNED
index_name-repo1-pull_request_review_comment-2020.10.24                           0     p      UNASSIGNED
index_name-repoexec-pull_request_review-2020.10.29                              0     r      UNASSIGNED
index_name-repoexec-pull_request_review-2020.10.29                              0     p      UNASSIGNED
index_name-repoexec-commit_comment                                              0     r      UNASSIGNED
index_name-repoexec-commit_comment                                              0     p      UNASSIGNED
index_name-repoexec-pull_request-2020.10.22                                     0     r      UNASSIGNED
index_name-repoexec-pull_request-2020.10.22                                     0     p      UNASSIGNED
index_name-repolib-pull_request_review-2020.10.28                               0     r      UNASSIGNED
index_name-repolib-pull_request_review-2020.10.28                               0     p      UNASSIGNED
index_name-repoexec-pull_request_review-2020.10.27                              0     r      UNASSIGNED
index_name-repoexec-pull_request_review-2020.10.27                              0     p      UNASSIGNED
index_name-repo1-issue_comment-2020.10.27                                         0     r      UNASSIGNED
index_name-repo1-issue_comment-2020.10.27                                         0     p      UNASSIGNED
testgithub-repo1-status                                                    0     p      UNASSIGNED
testgithub-repo1-status                                                    0     r      UNASSIGNED
index_name-repo1-issue_comment-2020.10.30                                         0     p      UNASSIGNED
index_name-repo1-issue_comment-2020.10.30                                         0     r      UNASSIGNED
index_name-%{[repository][name]}-%{[headers][x_github_event]}-2020.10.22        0     p      UNASSIGNED
index_name-%{[repository][name]}-%{[headers][x_github_event]}-2020.10.22        0     r      UNASSIGNED
index_name-repo1-pull_request-2020.10.26                                          0     p      UNASSIGNED
index_name-repo1-pull_request-2020.10.26                                          0     r      UNASSIGNED
index_name-repo1-pull_request                                                     0     p      UNASSIGNED
index_name-repo1-pull_request                                                     0     r      UNASSIGNED
index_name-repoexec-pull_request-2020.10.29                                     0     r      UNASSIGNED
index_name-repoexec-pull_request-2020.10.29                                     0     p      UNASSIGNED
.apm-custom-link                                                         0     r      UNASSIGNED
.apm-custom-link                                                         0     p      UNASSIGNED
index_name-repolib-status-2020.10.21                                            0     p      UNASSIGNED
index_name-repolib-status-2020.10.21                                            0     r      UNASSIGNED
index_name-repo2-pull_request                                               0     r      UNASSIGNED
index_name-repo2-pull_request                                               0     p      UNASSIGNED
index_name-repo4-issue_comment-2020.10.29                                        0     r      UNASSIGNED
index_name-repo4-issue_comment-2020.10.29                                        0     p      UNASSIGNED
index_name-repo2-team_add-2020.10.21                                        0     r      UNASSIGNED
index_name-repo2-team_add-2020.10.21                                        0     p      UNASSIGNED
index_name-repo4-push-2020.10.29                                                 0     r      UNASSIGNED
index_name-repo4-push-2020.10.29                                                 0     p      UNASSIGNED
index_name-repoexec-pull_request-2020.10.30                                     0     p      UNASSIGNED
index_name-repoexec-pull_request-2020.10.30                                     0     r      UNASSIGNED
index_name-repoexec-create-2020.10.30                                           0     p      UNASSIGNED
index_name-repoexec-create-2020.10.30                                           0     r      UNASSIGNED
index_name-repolib-pull_request_review-2020.10.22                               0     r      UNASSIGNED
index_name-repolib-pull_request_review-2020.10.22                               0     p      UNASSIGNED
index_name-testing_project_pipeline-%{[headers][x_github_event]}-2020.10.20     0     p      UNASSIGNED
index_name-testing_project_pipeline-%{[headers][x_github_event]}-2020.10.20     0     r      UNASSIGNED
index_name-repoexec-deploy_key-2020.10.20                                       0     r      UNASSIGNED
index_name-repoexec-deploy_key-2020.10.20                                       0     p      UNASSIGNED
index_name-repo4win-common-delete-2020.10.27                                     0     p      UNASSIGNED
index_name-repo4win-common-delete-2020.10.27                                     0     r      UNASSIGNED

Also there was another cluster/allocation/explain api which gave me the below error

root@node2:~# curl -XGET -u  localhost:9200/_cluster/allocation/explain
{"index":"test","shard":0,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"NODE_LEFT","at":"2020-11-18T09:41:34.834Z","details":"node_left [is8UsIpNR9GWX6LKG6J67w]","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"9nJaOObVRKuIxBzqi4ImCQ","node_name":"githublogging6","transport_address":"10.0.0.17:9300","node_attributes":{"ml.machine_memory":"4143378432","ml.max_open_jobs":"20","xpack.installed":"true","transform.node":"true"},"node_decision":"no","deciders":[{"decider":"replica_after_primary_active","decision":"NO","explanation":"primary shard for this replica is not yet active"},{"decider":"throttling","decision":"NO","explanation":"primary shard for this replica is not yet active"}]},{"node_id":"DbIXNuhlS7W1zbf9l-G_hw","node_name":"githublogging3","transport_address":"10.0.0.14:9300","node_attributes":{"ml.machine_memory":"4143378432","ml.max_open_jobs":"20","xpack.installed":"true","transform.node":"true"},"node_decision":"no","deciders":[{"decider":"replica_after_primary_active","decision":"NO","explanation":"primary shard for this replica is not yet active"},{"decider":"throttling","decision":"NO","explanation":"primary shard for this replica is not yet active"}]},{"node_id":"LOXyTVkIR5akOPRkh7fxVQ","node_name":"githublogging5","transport_address":"10.0.0.16:9300","node_attributes":{"ml.machine_memory":"4143378432","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"20"},"node_decision":"no","deciders":[{"decider":"replica_after_primary_active","decision":"NO","explanation":"primary shard for this replica is not yet active"},{"decider":"throttling","decision":"NO","explanation":"primary shard for this replica is not yet active"}]},{"node_id":"WBZYzxD4TtyIg1M9cu6P_Q","node_name":"githublogging2","transport_address":"10.0.0.20:9300","node_attributes":{"ml.machine_memory":"4143378432","ml.max_open_jobs":"20","xpack.installed":"true","transform.node":"true"},"node_decision":"no","deciders":[{"decider":"replica_after_primary_active","decision":"NO","explanation":"primary shard for this replica is not yet active"},{"decider":"throttling","decision":"NO","explanation":"primary shard for this replica is not yet active"}]},{"node_id":"beNfIHrHRqmsczDjMyPSDA","node_name":"githublogging4","transport_address":"10.0.0.15:9300","node_attributes":{"ml.machine_memory":"4143378432","ml.max_open_jobs":"20","xpack.installed":"true","transform.node":"true"},"node_decision":"no","deciders":[{"decider":"replica_after_primary_active","decision":"NO","explanation":"primary shard for this replica is not yet active"},{"decider":"throttling","decision":"NO","explanation":"primary shard for this replica is not yet active"}]}]}root@githubmaster2:~#

Also to be noted that this docker cluster comprises of 7 nodes. Is there any setting I should perform to fix this error? This is a production level environment.

Regards
TJ

2

What is the output from the _cluster/stats API?

3

Please find the output below

{
  "_nodes": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "cluster_name": "DevOps",
  "cluster_uuid": "QiItig0NRuS1DopvUABdgQ",
  "timestamp": 1606111643681,
  "status": "red",
  "indices": {
    "count": 440,
    "shards": {
      "total": 880,
      "primaries": 440,
      "replication": 1,
      "index": {
        "shards": {
          "min": 2,
          "max": 2,
          "avg": 2
        },
        "primaries": {
          "min": 1,
          "max": 1,
          "avg": 1
        },
        "replication": {
          "min": 1,
          "max": 1,
          "avg": 1
        }
      }
    },
    "docs": {
      "count": 29608717,
      "deleted": 5607073
    },
    "store": {
      "size_in_bytes": 30264054357,
      "reserved_in_bytes": 0
    },
    "fielddata": {
      "memory_size_in_bytes": 1657712,
      "evictions": 0
    },
    "query_cache": {
      "memory_size_in_bytes": 751786,
      "total_count": 7996879,
      "hit_count": 182430,
      "miss_count": 7814449,
      "cache_size": 214,
      "cache_count": 33417,
      "evictions": 33203
    },
    "completion": {
      "size_in_bytes": 0
    },
    "segments": {
      "count": 3954,
      "memory_in_bytes": 126872288,
      "terms_memory_in_bytes": 99969976,
      "stored_fields_memory_in_bytes": 2054656,
      "term_vectors_memory_in_bytes": 0,
      "norms_memory_in_bytes": 11772800,
      "points_memory_in_bytes": 0,
      "doc_values_memory_in_bytes": 13074856,
      "index_writer_memory_in_bytes": 63475184,
      "version_map_memory_in_bytes": 2848368,
      "fixed_bit_set_memory_in_bytes": 11326256,
      "max_unsafe_auto_id_timestamp": 1606089610187,
      "file_sizes": {}
    },
    "mappings": {
      "field_types": [
        {
          "name": "binary",
          "count": 14,
          "index_count": 3
        },
        {
          "name": "boolean",
          "count": 8939,
          "index_count": 472
        },
        {
          "name": "byte",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "date",
          "count": 4523,
          "index_count": 489
        },
        {
          "name": "date_nanos",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "date_range",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "double",
          "count": 8,
          "index_count": 8
        },
        {
          "name": "double_range",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "flattened",
          "count": 9,
          "index_count": 1
        },
        {
          "name": "float",
          "count": 75,
          "index_count": 10
        },
        {
          "name": "float_range",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "geo_point",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "geo_shape",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "half_float",
          "count": 78,
          "index_count": 22
        },
        {
          "name": "integer",
          "count": 200,
          "index_count": 18
        },
        {
          "name": "integer_range",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "ip",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "ip_range",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "keyword",
          "count": 25801,
          "index_count": 490
        },
        {
          "name": "long",
          "count": 11628,
          "index_count": 475
        },
        {
          "name": "long_range",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "nested",
          "count": 63,
          "index_count": 18
        },
        {
          "name": "object",
          "count": 8714,
          "index_count": 488
        },
        {
          "name": "shape",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "short",
          "count": 23,
          "index_count": 9
        },
        {
          "name": "text",
          "count": 24863,
          "index_count": 473
        }
      ]
    },
    "analysis": {
      "char_filter_types": [],
      "tokenizer_types": [],
      "filter_types": [
        {
          "name": "pattern_capture",
          "count": 1,
          "index_count": 1
        }
      ],
      "analyzer_types": [
        {
          "name": "custom",
          "count": 1,
          "index_count": 1
        }
      ],
      "built_in_char_filters": [],
      "built_in_tokenizers": [
        {
          "name": "uax_url_email",
          "count": 1,
          "index_count": 1
        }
      ],
      "built_in_filters": [
        {
          "name": "lowercase",
          "count": 1,
          "index_count": 1
        },
        {
          "name": "unique",
          "count": 1,
          "index_count": 1
        }
      ],
      "built_in_analyzers": []
    }
  },
  "nodes": {
    "count": {
      "total": 5,
      "coordinating_only": 0,
      "data": 5,
      "ingest": 5,
      "master": 5,
      "ml": 5,
      "remote_cluster_client": 5,
      "transform": 5,
      "voting_only": 0
    },
    "versions": [
      "7.9.1"
    ],
    "os": {
      "available_processors": 10,
      "allocated_processors": 10,
      "names": [
        {
          "name": "Linux",
          "count": 5
        }
      ],
      "pretty_names": [
        {
          "pretty_name": "CentOS Linux 7 (Core)",
          "count": 5
        }
      ],
      "mem": {
        "total_in_bytes": 20716892160,
        "free_in_bytes": 1112129536,
        "used_in_bytes": 19604762624,
        "free_percent": 5,
        "used_percent": 95
      }
    },
    "process": {
      "cpu": {
        "percent": 24
      },
      "open_file_descriptors": {
        "min": 1003,
        "max": 1031,
        "avg": 1011
      }
    },
    "jvm": {
      "max_uptime_in_millis": 2967958036,
      "versions": [
        {
          "version": "14.0.1",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "vm_version": "14.0.1+7",
          "vm_vendor": "AdoptOpenJDK",
          "bundled_jdk": true,
          "using_bundled_jdk": true,
          "count": 5
        }
      ],
      "mem": {
        "heap_used_in_bytes": 5219723104,
        "heap_max_in_bytes": 10737418240
      },
      "threads": 767
    },
    "fs": {
      "total_in_bytes": 2611971620864,
      "free_in_bytes": 2471502364672,
      "available_in_bytes": 2338703990784
    },
    "plugins": [],
    "network_types": {
      "transport_types": {
        "security4": 5
      },
      "http_types": {
        "security4": 5
      }
    },
    "discovery_types": {
      "zen": 5
    },
    "packaging_types": [
      {
        "flavor": "default",
        "type": "docker",
        "count": 5
      }
    ],
    "ingest": {
      "number_of_pipelines": 14,
      "processor_stats": {
        "conditional": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "geoip": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "grok": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "gsub": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "remove": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "rename": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "script": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        },
        "set": {
          "count": 0,
          "failed": 0,
          "current": 0,
          "time_in_millis": 0
        }
      }
    }
  }
}

I just realized that I have configured 7 node elasticsearch with 7 node(Docker Swarm) cluster but in the _cluster/stats it only shows 5.

ID                  NAME                     MODE                REPLICAS            IMAGE                                          PORTS
zke8gd5bh2ha        elastic_elasticsearch    global              7/7                 elasticsearch:7.9.1
npet3foigz26        elastic_kibana           replicated          1/1                 kibana:7.9.1
syu2is3lizuz        elastic_proxy            replicated          2/2                 dockerflow/docker-flow-proxy:latest            *:80->80/tcp, *:443->443/tcp, *:8200->8200/tcp, *:9200->9200/tcp
8ina06vrcujt        elastic_swarm-listener   replicated          1/1                 dockerflow/docker-flow-swarm-listener:latest

Does it have anything to do with the "Basic" License I reverted to recently?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.