The service was suddenly shutdown,then saw the logs"unexpected error while indexing monitoring document"

[KenChen (Chen)]

our elasticserach service was shutdown,we found "unexpected error while indexing monitoring document",then we searched many answers,but no one can resolve our problem.

 org.elasticsearch.xpack.monitoring.exporter.ExportException: UnavailableShardsException[[.monitoring-es-2-2018.06.28][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-2-2018.06.28][0]] containing [index {[.monitoring-es-2-2018.06.28][node_stats][AWRGNRZ39XlwqEQ75Mu_], source[{"cluster_uuid":"psiMFV9IQiqXfkhhMdH56w","timestamp":"2018-06-28T11:44:01.641Z","source_node":{"uuid":"J4KTs_cxQ0KxXGUTpC_g-Q","host":"10.10.201.90","transport_address":"10.10.201.90:9300","ip":"10.10.201.90","name":"esmaster01","attributes":{"rack":"esmaster01","ml.enabled":"true"}},"node_stats":{"node_id":"J4KTs_cxQ0KxXGUTpC_g-Q","node_master":false,"mlockall":false,"indices":{"docs":{"count":0},"store":{"size_in_bytes":0,"throttle_time_in_millis":0},"indexing":{"index_total":0,"index_time_in_millis":0,"throttle_time_in_millis":0},"search":{"query_total":0,"query_time_in_millis":0},"query_cache":{"memory_size_in_bytes":0,"hit_count":0,"miss_count":0,"evictions":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"segments":{"count":0,"memory_in_bytes":0,"terms_memory_in_bytes":0,"stored_fields_memory_in_bytes":0,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":0,"points_memory_in_bytes":0,"doc_values_memory_in_bytes":0,"index_writer_memory_in_bytes":0,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":0},"request_cache":{"memory_size_in_bytes":0,"evictions":0,"hit_count":0,"miss_count":0}},"os":{"cpu":{"load_average":{"1m":0.0,"5m":0.01,"15m":0.05}}},"process":{"open_file_descriptors":334,"max_file_descriptors":65536,"cpu":{"percent":0}},"jvm":{"mem":{"heap_used_in_bytes":2232433744,"heap_used_percent":29,"heap_max_in_bytes":7446462464},"gc":{"collectors":{"young":{"collection_count":43456,"collection_time_in_millis":1398788},"old":{"collection_count":3,"collection_time_in_millis":510}}}},"thread_pool":{"bulk":{"threads":8,"queue":0,"rejected":0},"generic":{"threads":34,"queue":0,"rejected":0},"get":{"threads":0,"queue":0,"rejected":0},"index":{"threads":0,"queue":0,"rejected":0},"management":{"threads":4,"queue":0,"rejected":0},"search":{"threads":13,"queue":0,"rejected":0},"watcher":{"threads":0,"queue":0,"rejected":0}},"fs":{"total":{"total_in_bytes":366499598336,"free_in_bytes":366437576704,"available_in_bytes":366437576704},"data":[{"spins":"true"}],"io_stats":{"total":{"operations":43232,"read_operations":93,"write_operations":43139,"read_kilobytes":372,"write_kilobytes":145632}}}}}]}]]]

    	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:131) ~[?:?]

    	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_131]
     org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:238) [elasticsearch-5.4.3.jar:5.4.3]
    Caused by: org.elasticsearch.action.UnavailableShardsException: [.monitoring-es-2-2018.06.28][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-2-2018.06.28][0]] containing [index {[.monitoring-es-2-2018.06.28][node_stats][AWRGNRZ39XlwqEQ75Mu_],    indent preformattePreformatted textd text by 4 spaces    indent preformatted text by 4 spaces    indent preformatted text by 4 spaces

[dadoonet (David Pilato)]

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

What gives:

GET /_cat/health?v