Setup mode is not available You do not have the necessary permissions to do this

Elastic Stack Kibana
22

Hi @Yogesh_Kumar1,

Could you run the following query against your monitoring cluster (Y.Y.Y.Y:9200) please? It's a slight variation of the query @chrisronline asked to run earlier. It will also show us when the monitoring data was last collected and into what indices:

POST .monitoring-es*/_search
{
  "size": 0,
  "aggs": {
    "by_cluster_uuid": {
      "terms": {
        "field": "cluster_uuid",
        "size": 10,
        "missing": "STANDALONE"
      },
      "aggs": {
        "by_doc_type": {
          "terms": {
            "field": "type",
            "size": 20
          },
          "aggs": {
            "by_index": {
              "terms": {
                "field": "_index",
                "size": 10
              },
              "aggs": {
                "latest_timestamp": {
                  "max": {
                    "field": "timestamp"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Also, could you please post the results of the following Elasticsearch API calls made against the same cluster?

GET _cat/indices/.monitoring-es*

and

GET _template/.monitoring-es

Thanks,

Shaunak

23

curl -XPOST http://Y.Y.Y.Y:9200/.monitoring-es*/_search?pretty -H 'Content-Type: application/json' -d'

{
"size": 0,
"aggs": {
"by_cluster_uuid": {
"terms": {
"field": "cluster_uuid",
"size": 10,
"missing": "STANDALONE"
},
"aggs": {
"by_doc_type": {
"terms": {
"field": "type",
"size": 20
},
"aggs": {
"by_index": {
"terms": {
"field": "_index",
"size": 10
},
"aggs": {
"latest_timestamp": {
"max": {
"field": "timestamp"
}
}
}
}
}
}
}
}
}
}'
{
"took" : 256,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 828,
"relation" : "eq"
},
"max_score" : null,
"hits" :
},
"aggregations" : {
"by_cluster_uuid" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "jNLcalI1SdSRpFJws2PqEw",
"doc_count" : 828,
"by_doc_type" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "node_stats",
"doc_count" : 828,
"by_index" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : ".monitoring-es-7-mb-2020.08.24",
"doc_count" : 828,
"latest_timestamp" : {
"value" : 1.598285705454E12,
"value_as_string" : "2020-08-24T16:15:05.454Z"
}
}
]
}
}
]
}
}
]
}
}
}

curl -XGET 'http://Y.Y.Y.Y:9200/_cat/indices/.monitoring-es*?pretty'
green open .monitoring-es-7-mb-2020.08.24 dO540sfYRWCreoFgyeTbdA 1 0 831 0 818.9kb 818.9kb

24

curl -XGET 'http://Y.Y.Y.Y:9200/_template/.monitoring-es'
{".monitoring-es":{"order":0,"version":7000199,"index_patterns":[".monitoring-es-7-*"],"settings":{"index":{"format":"7","codec":"best_compression","number_of_shards":"1","auto_expand_replicas":"0-1","number_of_replicas":"0"}},"mappings":{"date_detection":false,"dynamic":false,"properties":{"cluster_uuid":{"type":"keyword"},"state_uuid":{"type":"keyword"},"timestamp":{"type":"date","format":"date_time"},"interval_ms":{"type":"long"},"type":{"type":"keyword"},"source_node":{"properties":{"uuid":{"type":"keyword"},"host":{"type":"keyword"},"transport_address":{"type":"keyword"},"ip":{"type":"keyword"},"name":{"type":"keyword"},"timestamp":{"type":"date","format":"date_time"}}},"indices_stats":{"properties":{"_all":{"properties":{"primaries":{"properties":{"docs":{"properties":{"count":{"type":"long"}}},"indexing":{"properties":{"index_total":{"type":"long"},"index_time_in_millis":{"type":"long"}}},"search":{"properties":{"query_total":{"type":"long"},"query_time_in_millis":{"type":"long"}}}}},"total":{"properties":{"docs":{"properties":{"count":{"type":"long"}}},"indexing":{"properties":{"index_total":{"type":"long"},"index_time_in_millis":{"type":"long"}}},"search":{"properties":{"query_total":{"type":"long"},"query_time_in_millis":{"type":"long"}}}}}}}}},"index_stats":{"properties":{"index":{"type":"keyword"},"primaries":{"properties":{"docs":{"properties":{"count":{"type":"long"}}},"fielddata":{"properties":{"memory_size_in_bytes":{"type":"long"},"evictions":{"type":"long"}}},"store":{"properties":{"size_in_bytes":{"type":"long"}}},"indexing":{"properties":{"index_total":{"type":"long"},"index_time_in_millis":{"type":"long"},"throttle_time_in_millis":{"type":"long"}}},"merges":{"properties":{"total_size_in_bytes":{"type":"long"}}},"query_cache":{"properties":{"memory_size_in_bytes":{"type":"long"},"evictions":{"type":"long"},"hit_count":{"type":"long"},"miss_count":{"type":"long"}}},"request_cache":{"properties":{"memory_size_in_bytes":{"type":"long"},"evictions":{"type":"long"},"hit_count":{"type":"long"},"miss_count":{"type":"long"}}},"search":{"properties":{"query_total":{"type":"long"},"query_time_in_millis":{"type":"long"}}},"segments":{"properties":{"count":{"type":"integer"},"memory_in_bytes":{"type":"long"},"terms_memory_in_bytes":{"type":"long"},"points_memory_in_bytes":{"type":"long"},"stored_fields_memory_in_bytes":{"type":"long"},"term_vectors_memory_in_bytes":{"type":"long"},"norms_memory_in_bytes":{"type":"long"},"doc_values_memory_in_bytes":{"type":"long"},"index_writer_memory_in_bytes":{"type":"long"},"version_map_memory_in_bytes":{"type":"long"},"fixed_bit_set_memory_in_bytes":{"type":"long"}}},"refresh":{"properties":{"total_time_in_millis":{"type":"long"}}}}},"total":{"properties":{"docs":{"properties":{"count":{"type":"long"}}},"fielddata":{"properties":{"memory_size_in_bytes":{"type":"long"},"evictions":{"type":"long"}}},"store":{"properties":{"size_in_bytes":{"type":"long"}}},"indexing":{"properties":{"index_total":{"type":"long"},"index_time_in_millis":{"type":"long"},"throttle_time_in_millis":{"type":"long"}}},"merges":{"properties":{"total_size_in_bytes":{"type":"long"}}},"query_cache":{"properties":{"memory_size_in_bytes":{"type":"long"},"evictions":{"type":"long"},"hit_count":{"type":"long"},"miss_count":{"type":"long"}}},"request_cache":{"properties":{"memory_size_in_bytes":{"type":"long"},"evictions":{"type":"long"},"hit_count":{"type":"long"},"miss_count":{"type":"long"}}},"search":{"properties":{"query_total":{"type":"long"},"query_time_in_millis":{"type":"long"}}},"segments":{"properties":{"count":{"type":"integer"},"memory_in_bytes":{"type":"long"},"terms_memory_in_bytes":{"type":"long"},"points_memory_in_bytes":{"type":"long"},"stored_fields_memory_in_bytes":{"type":"long"},"term_vectors_memory_in_bytes":{"type":"long"},"norms_memory_in_bytes":{"type":"long"},"doc_values_memory_in_bytes":{"type":"long"},"index_writer_memory_in_bytes":{"type":"long"},"version_map_memory_in_bytes":{"type":"long"},"fixed_bit_set_memory_in_bytes":{"type":"long"}}},"refresh":{"properties":{"total_time_in_millis":{"type":"long"}}}}}}},"cluster_stats":{"properties":{"nodes":{"type":"object"},"indices":{"type":"object"}}},"cluster_state":{"properties":{"version":{"type":"long"},"nodes_hash":{"type":"integer"},"master_node":{"type":"keyword"},"state_uuid":{"type":"keyword"},"status":{"type":"keyword"},"nodes":{"type":"object"},"shards":{"type":"object"}}},"node_stats":{"properties":{"node_id":{"type":"keyword"},"node_master":{"type":"boolean"},"mlockall":{"type":"boolean"},"indices":{"properties":{"docs":{"properties":{"count":{"type":"long"}}},"fielddata":{"properties":{"memory_size_in_bytes":{"type":"long"},"evictions":{"type":"long"}}},"indexing":{"properties":{"index_time_in_millis":{"type":"long"},"index_total":{"type":"long"},"throttle_time_in_millis":{"type":"long"}}},"query_cache":{"properties":{"memory_size_in_bytes":{"type":"long"},"evictions":{"type":"long"},"hit_count":{"type":"long"},"miss_count":{"type":"long"}}},"request_cache":{"properties":{"memory_size_in_bytes":{"type":"long"},"evictions":{"type":"long"},"hit_count":{"type":"long"},"miss_count":{"type":"long"}}},"search":{"properties":{"query_time_in_millis":{"type":"long"},"query_total":{"type":"long"}}},"segments":{"properties":{"count":{"type":"integer"},"memory_in_bytes":{"type":"long"},"terms_memory_in_bytes":{"type":"long"},"points_memory_in_bytes":{"type":"long"},"stored_fields_memory_in_bytes":{"type":"long"},"term_vectors_memory_in_bytes":{"type":"long"},"norms_memory_in_bytes":{"type":"long"},"doc_values_memory_in_bytes":{"type":"long"},"index_writer_memory_in_bytes":{"type":"long"},"version_map_memory_in_bytes":{"type":"long"},"fixed_bit_set_memory_in_bytes":{"type":"long"}}},"store":{"properties":{"size_in_bytes":{"type":"long"}}}}},"fs":{"properties":{"total":{"properties":{"total_in_bytes":{"type":"long"},"free_in_bytes":{"type":"long"},"available_in_bytes":{"type":"long"}}},"data":{"properties":{"spins":{"type":"boolean"}}},"io_stats":{"properties":{"total":{"properties":{"operations":{"type":"long"},"read_operations":{"type":"long"},"write_operations":{"type":"long"},"read_kilobytes":{"type":"long"},"write_kilobytes":{"type":"long"}}}}}}},"os":{"properties":{"cgroup":{"properties":{"cpuacct":{"properties":{"control_group":{"type":"keyword"},"usage_nanos":{"type":"long"}}},"cpu":{"properties":{"cfs_quota_micros":{"type":"long"},"control_group":{"type":"keyword"},"stat":{"properties":{"number_of_elapsed_periods":{"type":"long"},"number_of_times_throttled":{"type":"long"},"time_throttled_nanos":{"type":"long"}}}}},"memory":{"properties":{"control_group":{"type":"keyword"},"limit_in_bytes":{"type":"keyword"},"usage_in_bytes":{"type":"keyword"}}}}},"cpu":{"properties":{"load_average":{"properties":{"1m":{"type":"half_float"},"5m":{"type":"half_float"},"15m":{"type":"half_float"}}}}}}},"process":{"properties":{"open_file_descriptors":{"type":"long"},"max_file_descriptors":{"type":"long"},"cpu":{"properties":{"percent":{"type":"half_float"}}}}},"jvm":{"properties":{"mem":{"properties":{"heap_used_in_bytes":{"type":"long"},"heap_used_percent":{"type":"half_float"},"heap_max_in_bytes":{"type":"long"}}},"gc":{"properties":{"collectors":{"properties":{"young":{"properties":{"collection_count":{"type":"long"},"collection_time_in_millis":{"type":"long"}}},"old":{"properties":{"collection_count":{"type":"long"},"collection_time_in_millis":{"type":"long"}}}}}}}}},"thread_pool":{"properties":{"bulk":{"properties":{"threads":{"type":"integer"},"queue":{"type":"integer"},"rejected":{"type":"long"}}},"generic":{"properties":{"threads":{"type":"integer"},"queue":{"type":"integer"},"rejected":{"type":"long"}}},"get":{"properties":{"threads":{"type":"integer"},"queue":{"type":"integer"},"rejected":{"type":"long"}}},"index":{"properties":{"threads":{"type":"integer"},"queue":{"type":"integer"},"rejected":{"type":"long"}}},"management":{"properties":{"threads":{"type":"integer"},"queue":{"type":"integer"},"rejected":{"type":"long"}}},"search":{"properties":{"threads":{"type":"integer"},"queue":{"type":"integer"},"rejected":{"type":"long"}}},"watcher":{"properties":{"threads":{"type":"integer"},"queue":{"type":"integer"},"rejected":{"type":"long"}}},"write":{"properties":{"queue":{"type":"integer"},"rejected":{"type":"long"}}}}}}},"index_recovery":{"type":"object"},"shard":{"properties":{"state":{"type":"keyword"},"primary":{"type":"boolean"},"index":{"type":"keyword"},"relocating_node":{"type":"keyword"},"shard":{"type":"long"},"node":{"type":"keyword"}}},"job_stats":{"properties":{"job_id":{"type":"keyword"},"state":{"type":"keyword"},"data_counts":{"properties":{"input_bytes":{"type":"long"},"processed_record_count":{"type":"long"},"empty_bucket_count":{"type":"long"},"sparse_bucket_count":{"type":"long"},"bucket_count":{"type":"long"},"earliest_record_timestamp":{"type":"date"},"latest_record_timestamp":{"type":"date"}}},"model_size_stats":{"properties":{"model_bytes":{"type":"long"},"bucket_allocation_failures_count":{"type":"long"}}},"node":{"properties":{"id":{"type":"keyword"}}}}},"ccr_stats":{"properties":{"remote_cluster":{"type":"keyword"},"leader_index":{"type":"keyword"},"follower_index":{"type":"keyword"},"shard_id":{"type":"integer"},"leader_global_checkpoint":{"type":"long"},"leader_max_seq_no":{"type":"long"},"follower_global_checkpoint":{"type":"long"},"follower_max_seq_no":{"type":"long"},"last_requested_seq_no":{"type":"long"},"outstanding_read_requests":{"type":"long"},"outstanding_write_requests":{"type":"long"},"write_buffer_operation_count":{"type":"long"},"write_buffer_size_in_bytes":{"type":"long"},"follower_mapping_version":{"type":"long"},"follower_settings_version":{"type":"long"},"follower_aliases_version":{"type":"long"},"total_read_time_millis":{"type":"long"},"total_read_remote_exec_time_millis":{"type":"long"},"successful_read_requests":{"type":"long"},"failed_read_requests":{"type":"long"},"operations_read":{"type":"long"},"bytes_read":{"type":"long"},"total_write_time_millis":{"type":"long"},"successful_write_requests":{"type":"long"},"failed_write_requests":{"type":"long"},"operations_written":{"type":"long"},"read_exceptions":{"type":"nested","properties":{"from_seq_no":{"type":"long"},"retries":{"type":"integer"},"exception":{"type":"object","properties":{"type":{"type":"keyword"},"reason":{"type":"text"}}}}},"time_since_last_read_millis":{"type":"long"},"fatal_exception":{"type":"object","properties":{"type":{"type":"keyword"},"reason":{"type":"text"}}}}},"ccr_auto_follow_stats":{"properties":{"number_of_failed_follow_indices":{"type":"long"},"number_of_failed_remote_cluster_state_requests":{"type":"long"},"number_of_successful_follow_indices":{"type":"long"},"recent_auto_follow_errors":{"type":"nested","properties":{"leader_index":{"type":"keyword"},"timestamp":{"type":"long"},"auto_follow_exception":{"type":"object","properties":{"type":{"type":"keyword"},"reason":{"type":"text"}}}}},"auto_followed_clusters":{"type":"nested","properties":{"cluster_name":{"type":"keyword"},"time_since_last_check_millis":{"type":"long"},"last_seen_metadata_version":{"type":"long"}}}}},"enrich_coordinator_stats":{"properties":{"node_id":{"type":"keyword"},"queue_size":{"type":"integer"},"remote_requests_current":{"type":"long"},"remote_requests_total":{"type":"long"},"executed_searches_total":{"type":"long"}}},"enrich_executing_policy_stats":{"properties":{"name":{"type":"keyword"},"task":{"type":"object","properties":{"node":{"type":"keyword"},"id":{"type":"long"},"type":{"type":"keyword"},"action":{"type":"keyword"},"description":{"type":"keyword"},"start_time_in_millis":{"type":"date","format":"epoch_millis"},"running_time_in_nanos":{"type":"long"},"cancellable":{"type":"boolean"}}}}}}},"aliases":{}}}

25

Thanks for the responses. This confirms that the Elasticsearch monitoring data is being collected by Metricbeat and is up to date in the monitoring indices. It also confirms that the monitoring index template exists and looks good.

At this point, I'd suggest the following next steps as a temporary measure for debugging the issue:

  1. Stop Metricbeat.

  2. Disable the system module, just to reduce some noise in the logs.

    metricbeat modules disable system

  3. Restart Metricbeat with debug-level logging. This is expected to be quite verbose.

    metricbeat -e -d'*'

  4. Post the logs from the previous step, right from the beginning of the log to about 20 seconds in. Feel free to use pastebin.com or gist.github.com if the logs are too much to post over here directly. If you do post them here directly, please enclose them in ``` delimiters so they are properly formatted and easy to read.

  5. Stop Metricbeat.

Thanks,

Shaunak

26

Here are the debug logs of metricbeat :
metricbeat logs

27

Thanks @Yogesh_Kumar1 for the debug logs. Looking at them, I see messages like this:

Starting metricSetWrapper[module=elasticsearch, name=...

There are 9 such messages, one for each elasticsearch module metricset that is automatically enabled when xpack.enabled: true is set. So far so good.

Further down, I see messages like this:

trying to fetch ... from a non-master node

There are 8 such messages, corresponding to 8 of the 9 metricsets, except for node_stats.

And finally, we see an event being published by the node_stats metricset.

Now, looking at your modules.d/elasticsearch.yml configuration, I see this:

hosts: ["http://X.X.X.X2:9200"]

This is where I think we need a change. In your original post you mentioned you have an Elasticsearch cluster of 5 nodes. You will need to add the addresses of each of these 5 nodes to the hosts setting, something like this:

hosts:
  - "http://X.X.X.X1:9200"
  - "http://X.X.X.X2:9200"
  - "http://X.X.X.X3:9200"
  - "http://X.X.X.X4:9200"
  - "http://X.X.X.X5:9200"

The reason for this is that the elasticsearch module in Metricbeat 7.8.1 and even 7.9.0 needs to collect monitoring data from every Elasticsearch node in the cluster.

In 7.10, we've introduced a new setting named scope in the elasticsearch module configuration - this setting will let you point the module at any node in the Elasticsearch cluster.

For now, please update your hosts setting as I explained above and let us know if that helps.

Cheers,

Shaunak

28

you nailed it man :slight_smile:
It worked . thanks @shaunak and @chrisronline for the help

29

one quick question why this dashboard is showing data in bytes instead of GB?

Screenshot from 2020-08-26 19-17-54
Screenshot from 2020-08-26 19-17-541261×532 59 KB

30

Known bug here : https://github.com/elastic/kibana/issues/73298

31

thanks @chrisronline

32

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.