This is my elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: []
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
ingest.geoip.downloader.enabled: false
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 17-11-2022 09:55:02
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["ip-10-0-9-223.ap-south-1.compute.internal"]
discovery.type: single-node
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
This is kibana.yml
# For more configuration options see the configuration guide for Kibana in
# https://www.elastic.co/guide/index.html
# =================== System: Kibana Server ===================
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "0.0.0.0"
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# Defaults to `false`.
#server.rewriteBasePath: false
# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""
# The maximum payload size in bytes for incoming server requests.
#server.maxPayload: 1048576
# The Kibana server's name. This is used for display purposes.
#server.name: "your-hostname"
# =================== System: Kibana Server (Optional) ===================
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# =================== System: Elasticsearch ===================
# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "elastic"
#elasticsearch.password: "minutus"
# Kibana can also authenticate to Elasticsearch via "service account tokens".
# Service account tokens are Bearer style tokens that replace the traditional username/password based configuration.
# Use this token instead of a username/password.
# elasticsearch.serviceAccountToken: "my_token"
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# The maximum number of sockets that can be used for communications with elasticsearch.
# Defaults to `Infinity`.
#elasticsearch.maxSockets: 1024
# Specifies whether Kibana should use compression for communications with elasticsearch
# Defaults to `false`.
#elasticsearch.compression: false
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# =================== System: Elasticsearch (Optional) ===================
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
# =================== System: Logging ===================
# Set the value of this setting to off to suppress all logging output, or to debug to log everything. Defaults to 'info'
#logging.root.level: debug
# Enables you to specify a file where Kibana stores log output.
logging:
appenders:
file:
type: file
fileName: /var/log/kibana/kibana.log
layout:
type: json
root:
appenders:
- default
- file
# layout:
# type: json
# Logs queries sent to Elasticsearch.
#logging.loggers:
# - name: elasticsearch.query
# level: debug
# Logs http responses.
#logging.loggers:
# - name: http.server.response
# level: debug
# Logs system usage information.
#logging.loggers:
# - name: metrics.ops
# level: debug
# =================== System: Other ===================
# The path where Kibana stores persistent data not saved in Elasticsearch. Defaults to data
#path.data: data
# Specifies the path where Kibana creates the process ID file.
pid.file: /run/kibana/kibana.pid
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000ms.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English (default) "en", Chinese "zh-CN", Japanese "ja-JP", French "fr-FR".
#i18n.locale: "en"
# =================== Frequently used (Optional)===================
# =================== Saved Objects: Migrations ===================
# Saved object migrations run at startup. If you run into migration-related issues, you might need to adjust these settings.
# The number of documents migrated at a time.
# If Kibana can't start up or upgrade due to an Elasticsearch `circuit_breaking_exception`,
# use a smaller batchSize value to reduce the memory pressure. Defaults to 1000 objects per batch.
#migrations.batchSize: 1000
# The maximum payload size for indexing batches of upgraded saved objects.
# To avoid migrations failing due to a 413 Request Entity Too Large response from Elasticsearch.
# This value should be lower than or equal to your Elasticsearch cluster’s `http.max_content_length`
# configuration option. Default: 100mb
#migrations.maxBatchSizeBytes: 100mb
# The number of times to retry temporary migration failures. Increase the setting
# if migrations fail frequently with a message such as `Unable to complete the [...] step after
# 15 attempts, terminating`. Defaults to 15
#migrations.retryAttempts: 15
# =================== Search Autocomplete ===================
# Time in milliseconds to wait for autocomplete suggestions from Elasticsearch.
# This value must be a whole number greater than zero. Defaults to 1000ms
#unifiedSearch.autocomplete.valueSuggestions.timeout: 1000
# Maximum number of documents loaded by each shard to generate autocomplete suggestions.
# This value must be a whole number greater than zero. Defaults to 100_000
#unifiedSearch.autocomplete.valueSuggestions.terminateAfter: 100000
# This section was automatically generated during setup.
elasticsearch.hosts: ['http://10.0.9.223:9200']
elasticsearch.serviceAccountToken: AAEAAWVsYXN0aWMva2liYW5hL2Vucm9sbC1wcm9jZXNzLXRva2VuLTE2Njg2ODAyNzY1NjA6QU9sZjU3UlBTNTJRcFpmZnZ4TG9tQQ
elasticsearch.ssl.certificateAuthorities: [/var/lib/kibana/ca_1668680277725.crt]
xpack.fleet.outputs: [{id: fleet-default-output, name: default, is_default: true, is_default_monitoring: true, type: elasticsearch, hosts: ['http://10.0.9.223:9200'], ca_trusted_fingerprint: 44ff87aa97797b9aff6cad49f71a4f04a99743f8b46922fee8e84babd8223efa}]
This is the log
tail -f /var/log/elasticsearch/elasticsearch.log
[2022-11-21T06:28:32,385][INFO ][o.e.c.s.ClusterApplierService] [ip-10-0-9-223.ap-south-1.compute.internal] master node changed {previous [], current [{ip-10-0-9-223.ap-south-1.compute.internal}{7npnQ-SzRfWswbi2HuCbmw}{mxXO8y3JSve3h9ESYIs7hg}{ip-10-0-9-223.ap-south-1.compute.internal}{10.0.9.223}{10.0.9.223:9300}{cdfhilmrstw}]}, term: 29, version: 892, reason: Publication{term=29, version=892}
[2022-11-21T06:28:32,465][INFO ][o.e.r.s.FileSettingsService] [ip-10-0-9-223.ap-south-1.compute.internal] starting file settings watcher ...
[2022-11-21T06:28:32,507][INFO ][o.e.h.AbstractHttpServerTransport] [ip-10-0-9-223.ap-south-1.compute.internal] publish_address {10.0.9.223:9200}, bound_addresses {[::]:9200}
[2022-11-21T06:28:32,508][INFO ][o.e.n.Node ] [ip-10-0-9-223.ap-south-1.compute.internal] started {ip-10-0-9-223.ap-south-1.compute.internal}{7npnQ-SzRfWswbi2HuCbmw}{mxXO8y3JSve3h9ESYIs7hg}{ip-10-0-9-223.ap-south-1.compute.internal}{10.0.9.223}{10.0.9.223:9300}{cdfhilmrstw}{ml.max_jvm_size=4102029312, ml.allocated_processors=2, ml.machine_memory=8199434240, xpack.installed=true, ml.allocated_processors_double=2.0}
[2022-11-21T06:28:32,497][INFO ][o.e.r.s.FileSettingsService] [ip-10-0-9-223.ap-south-1.compute.internal] file settings service up and running [tid=51]
[2022-11-21T06:28:32,866][INFO ][o.e.l.LicenseService ] [ip-10-0-9-223.ap-south-1.compute.internal] license [2131957f-667a-421e-95ab-585fc0a1dd2e] mode [basic] - valid
[2022-11-21T06:28:32,881][INFO ][o.e.x.s.a.Realms ] [ip-10-0-9-223.ap-south-1.compute.internal] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-11-21T06:28:32,891][INFO ][o.e.g.GatewayService ] [ip-10-0-9-223.ap-south-1.compute.internal] recovered [21] indices into cluster_state
[2022-11-21T06:28:32,964][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [ip-10-0-9-223.ap-south-1.compute.internal] Node [{ip-10-0-9-223.ap-south-1.compute.internal}{7npnQ-SzRfWswbi2HuCbmw}] is selected as the current health node.
[2022-11-21T06:28:35,312][INFO ][o.e.c.r.a.AllocationService] [ip-10-0-9-223.ap-south-1.compute.internal] current.health="YELLOW" message="Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.ds-metricbeat-8.4.2-2022.11.17-000001][0]]])." previous.health="RED" reason="shards started [[.ds-metricbeat-8.4.2-2022.11.17-000001][0]]"
[2022-11-21T06:29:49,573][INFO ][o.e.c.m.MetadataDeleteIndexService] [ip-10-0-9-223.ap-south-1.compute.internal] [kibana_syslog2022.11.18/j28bnE6LTSODkOLd7QkrPQ] deleting index
[2022-11-21T06:29:49,574][INFO ][o.e.c.m.MetadataDeleteIndexService] [ip-10-0-9-223.ap-south-1.compute.internal] [elasticsearch_syslog2022.11.18/GkEheWhlSYC60fTfPPz_PA] deleting index
[2022-11-21T06:29:49,574][INFO ][o.e.c.m.MetadataDeleteIndexService] [ip-10-0-9-223.ap-south-1.compute.internal] [elasticsearch_syslog2022.11.21/puqt2R1ZQraE7pKJ2E8p3w] deleting index
[2022-11-21T06:29:49,575][INFO ][o.e.c.m.MetadataDeleteIndexService] [ip-10-0-9-223.ap-south-1.compute.internal] [jenkins_syslog2022.11.18/B3F1KpHtTkiawf4tnfyjZA] deleting index
[2022-11-21T06:29:49,575][INFO ][o.e.c.m.MetadataDeleteIndexService] [ip-10-0-9-223.ap-south-1.compute.internal] [logstash_syslog2022.11.21/9rmjjba1RVKl0KRf6BjzPw] deleting index
[2022-11-21T06:29:49,575][INFO ][o.e.c.m.MetadataDeleteIndexService] [ip-10-0-9-223.ap-south-1.compute.internal] [logstash_syslog2022.11.18/vy_jPuIBSgu3hUHo82D7Vw] deleting index
[2022-11-21T06:29:49,575][INFO ][o.e.c.m.MetadataDeleteIndexService] [ip-10-0-9-223.ap-south-1.compute.internal] [kibana_syslog2022.11.21/XRMk_LW3Q7yypjAZGXOJ7g] deleting index
[2022-11-21T06:29:50,412][INFO ][o.e.c.m.MetadataCreateIndexService] [ip-10-0-9-223.ap-south-1.compute.internal] [logstash_syslog2022.11.21] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2022-11-21T06:29:50,694][INFO ][o.e.c.m.MetadataMappingService] [ip-10-0-9-223.ap-south-1.compute.internal] [logstash_syslog2022.11.21/VrCJA3khRKWFy7y5Lx4O7Q] create_mapping
[2022-11-21T06:29:54,901][INFO ][o.e.c.m.MetadataCreateIndexService] [ip-10-0-9-223.ap-south-1.compute.internal] [elasticsearch_syslog2022.11.21] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2022-11-21T06:29:55,053][INFO ][o.e.c.m.MetadataMappingService] [ip-10-0-9-223.ap-south-1.compute.internal] [elasticsearch_syslog2022.11.21/HW7znSaCRHmX7rcbLjqxig] create_mapping
[2022-11-21T06:30:23,086][INFO ][o.e.c.m.MetadataMappingService] [ip-10-0-9-223.ap-south-1.compute.internal] [logstash_syslog2022.11.21/VrCJA3khRKWFy7y5Lx4O7Q] update_mapping [_doc]
[2022-11-21T06:31:17,279][WARN ][o.e.t.ThreadPool ] [ip-10-0-9-223.ap-south-1.compute.internal] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@78ef6630, interval=5s}] took [43412ms] which is above the warn threshold of [5000ms]
[2022-11-21T06:31:36,261][WARN ][o.e.t.ThreadPool ] [ip-10-0-9-223.ap-south-1.compute.internal] execution of [ReschedulingRunnable{runnable=org.elasticsearch.monitor.jvm.JvmGcMonitorService$1@58abcf40, interval=1s}] took [15397ms] which is above the warn threshold of [5000ms]
[2022-11-21T06:31:57,318][WARN ][o.e.t.ThreadPool ] [ip-10-0-9-223.ap-south-1.compute.internal] execution of [ReschedulingRunnable{runnable=org.elasticsearch.indices.IndexingMemoryController$ShardsIndicesStatusChecker@7638b22e, interval=5s}] took [19015ms] which is above the warn threshold of [5000ms]
[2022-11-21T06:32:12,134][WARN ][o.e.h.AbstractHttpServerTransport] [ip-10-0-9-223.ap-south-1.compute.internal] handling request [unknownId][POST][/.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true][Netty4HttpChannel{localAddress=/10.0.9.223:9200, remoteAddress=/10.0.9.223:54476}] took [34685ms] which is above the warn threshold of [5000ms]
[2022-11-21T06:32:35,251][WARN ][o.e.t.ThreadPool ] [ip-10-0-9-223.ap-south-1.compute.internal] execution of [ReschedulingRunnable{runnable=org.elasticsearch.monitor.jvm.JvmGcMonitorService$1@58abcf40, interval=1s}] took [6654ms] which is above the warn threshold of [5000ms]
[2022-11-21T06:32:35,386][WARN ][o.e.t.ThreadPool ] [ip-10-0-9-223.ap-south-1.compute.internal] timer thread slept for [6.6s/6625ms] on absolute clock which is above the warn threshold of [5000ms]
[2022-11-21T06:32:38,710][WARN ][o.e.t.ThreadPool ] [ip-10-0-9-223.ap-south-1.compute.internal] timer thread slept for [6.6s/6654209857ns] on relative clock which is above the warn threshold of [5000ms]
[2022-11-21T06:32:52,172][WARN ][o.e.t.ThreadPool ] [ip-10-0-9-223.ap-south-1.compute.internal] execution of [ReschedulingRunnable{runnable=org.elasticsearch.monitor.jvm.JvmGcMonitorService$1@58abcf40, interval=1s}] took [10072ms] which is above the warn threshold of [5000ms]
When i start elasticsearch and kibana, the services are active. i am able to login to kibana. but after a couple of minutes the server hangs up and elasticsearch service fails. nothing shows us in status elasticsearch -l.