Error in logs of master and data-ingest nodes

I deployed Elasticsearch cluster in GCP. Using ECK: 1.3.0 and elasticsearch and kibana as 7.9.0. All pods are running fine. But when I checked the logs of data-ingest and master nodes. I can see below error.

{"type": "server", "timestamp": "2020-12-06T14:25:31,046Z", "level": "ERROR", "component": "o.e.x.s.a.e.NativeUsersStore", "cluster.name": "elasticsearch-config", "node.name": "elasticsearch-config-es-master-0", "message": "security index is unavailable. short circuiting retrieval of user [elastic-internal]", "cluster.uuid": "s0FRSzogSgqHXXyWLeQJEA", "node.id": "UnoMZ2tjTrukpZPfRbgDCg"  }

Why getting this error?

What is the output from the _cat/indices?v endpoint?

Below is the output that I am getting

health status index                               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
red    open   .kibana-event-log-7.9.0-000001      _FwSBFekTeinyeZ_6F4a1w   1   0                                              
red    open   .apm-agent-configuration            QQADQtwBS1OTO7kOMmsyig   1   0                                              
red    open   packetbeat-7.9.2-2020.12.07-000001  KxzVMhjPTbKZYtvh5WDfJQ   1   1                                              
red    open   .kibana_1                           f4wAQ3noQRyjdIJvtMGGmA   1   0                                              
red    open   metricbeat-7.9.2-2020.12.07-000001  OAMN72HtQqKxKgD3x3vOBQ   1   1                                              
red    open   .security-7                         xYqC6AkqT1CYJxSnAcaGUA   1   0                                              
red    open   journalbeat-7.9.2-2020.12.07-000001 cbI6tjO8TriAoqsuJAuMlQ   1   1                                              
red    open   .apm-custom-link                    b5qeBZheS1yLJKbRqLu1eg   1   0                                              
red    open   index002                            S-zEt-YdRtWLfx71aYKp7w   1   1                                              
red    open   .kibana_task_manager_1              na7xEQq8TAeSBOb6S2HDFQ   1   0                                              
red    open   auditbeat-7.9.2-2020.12.07-000001   Vq99cfxBQJ-49Ih_R4gQpw   1   1                                              
red    open   index-001                           9F6T5-F7T6yzJlsehu5o_Q   1   1                                              
red    open   heartbeat-7.9.2-2020.12.07-000001   98MGTBanSjSW_loOW_KueA   1   1                                              

You've got a few issues with your cluster, every index is red.

What is the output from _cluster/stats?

Below is the output

{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"elasticsearch-config","cluster_uuid":"HOmHTXwORSWeJ9VMPBQxkQ","timestamp":1607320736637,"status":"red","indices":{"count":0,"shards":{},"docs":{"count":0,"deleted":0},"store":{"size_in_bytes":0,"reserved_in_bytes":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"query_cache":{"memory_size_in_bytes":0,"total_count":0,"hit_count":0,"miss_count":0,"cache_size":0,"cache_count":0,"evictions":0},"completion":{"size_in_bytes":0},"segments":{"count":0,"memory_in_bytes":0,"terms_memory_in_bytes":0,"stored_fields_memory_in_bytes":0,"term_vectors_memory_in_bytes":0,"norms_memory_in_bytes":0,"points_memory_in_bytes":0,"doc_values_memory_in_bytes":0,"index_writer_memory_in_bytes":0,"version_map_memory_in_bytes":0,"fixed_bit_set_memory_in_bytes":0,"max_unsafe_auto_id_timestamp":-9223372036854775808,"file_sizes":{}},"mappings":{"field_types":[{"name":"alias","count":72,"index_count":2},{"name":"binary","count":13,"index_count":2},{"name":"boolean","count":208,"index_count":9},{"name":"byte","count":1,"index_count":1},{"name":"date","count":202,"index_count":12},{"name":"date_range","count":1,"index_count":1},{"name":"double","count":98,"index_count":1},{"name":"flattened","count":9,"index_count":1},{"name":"float","count":130,"index_count":6},{"name":"geo_point","count":36,"index_count":5},{"name":"integer","count":31,"index_count":3},{"name":"ip","count":88,"index_count":5},{"name":"keyword","count":4206,"index_count":12},{"name":"long","count":2696,"index_count":8},{"name":"nested","count":14,"index_count":4},{"name":"object","count":3239,"index_count":12},{"name":"scaled_float","count":112,"index_count":1},{"name":"text","count":395,"index_count":11}]},"analysis":{"char_filter_types":[],"tokenizer_types":[],"filter_types":[{"name":"pattern_capture","count":1,"index_count":1}],"analyzer_types":[{"name":"custom","count":1,"index_count":1}],"built_in_char_filters":[],"built_in_tokenizers":[{"name":"uax_url_email","count":1,"index_count":1}],"built_in_filters":[{"name":"lowercase","count":1,"index_count":1},{"name":"unique","count":1,"index_count":1}],"built_in_analyzers":[{"name":"simple","count":4,"index_count":1}]}},"nodes":{"count":{"total":1,"coordinating_only":0,"data":0,"ingest":0,"master":1,"ml":1,"remote_cluster_client":1,"transform":0,"voting_only":0},"versions":["7.9.0"],"os":{"available_processors":1,"allocated_processors":1,"names":[{"name":"Linux","count":1}],"pretty_names":[{"pretty_name":"CentOS Linux 7 (Core)","count":1}],"mem":{"total_in_bytes":3221225472,"free_in_bytes":595521536,"used_in_bytes":2625703936,"free_percent":18,"used_percent":82}},"process":{"cpu":{"percent":0},"open_file_descriptors":{"min":430,"max":430,"avg":430}},"jvm":{"max_uptime_in_millis":75149908,"versions":[{"version":"14.0.1","vm_name":"OpenJDK 64-Bit Server VM","vm_version":"14.0.1+7","vm_vendor":"AdoptOpenJDK","bundled_jdk":true,"using_bundled_jdk":true,"count":1}],"mem":{"heap_used_in_bytes":1336735744,"heap_max_in_bytes":2147483648},"threads":45},"fs":{"total_in_bytes":315993423872,"free_in_bytes":315925983232,"available_in_bytes":315909206016},"plugins":[{"name":"repository-gcs","version":"7.9.0","elasticsearch_version":"7.9.0","java_version":"1.8","description":"The GCS repository plugin adds Google Cloud Storage support for repositories.","classname":"org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin","extended_plugins":[],"has_native_controller":false}],"network_types":{"transport_types":{"security4":1},"http_types":{"security4":1}},"discovery_types":{"zen":1},"packaging_types":[{"flavor":"default","type":"docker","count":1}],"ingest":{"number_of_pipelines":1,"processor_stats":{"gsub":{"count":0,"failed":0,"current":0,"time_in_millis":0},"script":{"count":0,"failed":0,"current":0,"time_in_millis":0}}}}}

My elasticsearch cluster health becomes red when I added beats part.

Your description implies a multi-node cluster, but the cluster output shows only 1.
Something's not right here.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.