Template for .monitoring* indices

Elastic Stack Elasticsearch
1

Hi,
In my setup I am using template from Logstash to set the configuration for various indices that are being created in Elasticsearch. But I am not able to do the same for the monitoring indices, that is .monitoring-es*, .monitoring-logstash*, etc.
I found that I can set them with REST calls but I would like the settings to be applied automatically without executing any queries. Need help for the same.

Thank you in advance

2

Why do you want to change those template?

3

I would like to increase the replica count. This is mainly because there are multiple occasions where the shards are failing over a restart and the cluster is in unusable state

4

Why does the cluster get into an unusable state? How many master-eligible nodes do you have in the cluster? What is discovery.zen.minimum_master_nodes set to?

5

Master eligible nodes are 2. (I know this setting is to be an odd number)
I am getting error like below

.monitoring-logstash-6-2018.12.12/PJ98ndmCTl-R1c1qUTGpng]] can not be imported as a dangling index, as an index with the same name and UUID exist in the index tombstones. This situation is likely caused by copying over the data directory for an index that was previously deleted.

(DATA folder was not copied or deleted. Only restarted )

org.elasticsearch.action.UnavailableShardsException: [.monitoring-es-6-2018.12.20][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-6-2018.12.20][0]] containing [index {[.monitoring-es-6-2018.12.20][doc][MrzYymcBNIueI7r1YAIm], source[{"cluster_uuid":"lqoa3HJhRPyUAQUkLgFeqw","timestamp":"2018-12-20T09:00:37.283Z","interval_ms":10000,"type":"node_stats","source_node":{"uuid":"LdsS8QO7SUauL7vUOYljMw","host":"xxxxxxx","transport_address":"xxxxxxxx","ip":"xxxxxx","name":"elklog02","timestamp":"2018-12-20T09:00:37.283Z"},"node_stats":{"node_id":"LdsS8QO7SUauL7vUOYljMw","node_master":false,"mlockall":false,"indices":{"docs":{"count":0},"store":{"size_in_bytes":0},"indexing":{"index_total":0,"index_time_in_millis":0,"throttle_time_in_millis":0},"search":{"query_total":0,"query_time_in_millis":0},"query_cache":{"memory_size_in_bytes":0,"hit_count":0,"miss_count":0,"evictions":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"segments":{"count":0,"memory_in_bytes":0,"terms_memory_in_bytes":0,"stored_fields_memory_in_bytes":0,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":0,"points_memory_in_bytes":0,"doc_values_memory_in_bytes":0,"index_writer_memory_in_bytes":0,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":0},"request_cache":{"memory_size_in_bytes":0,"evictions":0,"hit_count":0,"miss_count":0}},"os":{"cpu":{"load_average":{"1m":0.06,"5m":0.27,"15m":0.47}},"cgroup":{"cpuacct":{"control_group":"/","usage_nanos":329135454114692},"cpu":{"control_group":"/","cfs_period_micros":100000,"cfs_quota_micros":-1,"stat":{"number_of_elapsed_periods":0,"number_of_times_throttled":0,"time_throttled_nanos":0}},"memory":{"control_group":"/","limit_in_bytes":"9223372036854771712","usage_in_bytes":"7188840448"}}},"process":{"open_file_descriptors":353,"max_file_descriptors":65536,"cpu":{"percent":1}},"jvm":{"mem":{"heap_used_in_bytes":883829600,"heap_used_percent":27,"heap_max_in_bytes":3203792896},"gc":{"collectors":{"young":{"collection_count":218,"collection_time_in_millis":7084},"old":{"collection_count":2,"collection_time_in_millis":88}}}},"thread_pool":{"generic":{"threads":53,"queue":0,"rejected":0},"get":{"threads":0,"queue":0,"rejected":0},"index":{"threads":0,"queue":0,"rejected":0},"management":{"threads":2,"queue":0,"rejected":0},"search":{"threads":0,"queue":0,"rejected":0},"watcher":{"threads":0,"queue":0,"rejected":0},"write":{"threads":0,"queue":0,"rejected":0}},"fs":{"total":{"total_in_bytes":50986217472,"free_in_bytes":45045489664,"available_in_bytes":45045489664},"io_stats":{"total":{"operations":43899,"read_operations":33705,"write_operations":10194,"read_kilobytes":1088276,"write_kilobytes":147328}}}}}]}]]

Also please point me to setting the monitoring indices template as well.

6

This is indeed not ideal. If you want a highly available cluster it has to be at least 3. Is discovery.zen.minimum_master_nodes set correctly according to these guidelines?

7

I did it after posting the previous reply, Thank you. Am I correct to say that dangling index is generally caused by split brain?
But why UnavailableShardsException is caused. Can you please provide explanation for this.

8

I have the min master node count set but I am still getting the below exception on restart

org.elasticsearch.action.UnavailableShardsException: [.monitoring-es-6-2018.12.20][0] primary shard is not active Timeout: [1m]

9

I got the answer for shards being unavailable. This was because the cluster.routing.allocation.node_initial_primaries_recoveries was as default value of 2. Hence it took time to recover the primary shard. Increasing this solved the above mentioned issue.
I got this answer by executing the below query

GET /_cluster/allocation/explain
{
  "index": ".monitoring-es-6-2018.12.20",
  "shard": 0,
  "primary": true
}

But I still do not have answer regarding how the template can be set for the monitoring indices.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.