Failed to start Elasticsearch 7.11.2

Nik1 · May 30, 2023, 12:03pm

Elasticsearch service is not starting on the ubuntu machine. Getting the below error. Anybody having idea?

~# systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

~# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2023-05-30 11:56:03 UTC; 21s ago
Docs: https://www.elastic.co
Process: 10349 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 10349 (code=exited, status=1/FAILURE)

systemd-entrypoint[10349]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
systemd-entrypoint[10349]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
systemd-entrypoint[10349]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
systemd-entrypoint[10349]: at org.elasticsearch.cli.Command.main(Command.java:79)
systemd-entrypoint[10349]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
systemd-entrypoint[10349]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
systemd-entrypoint[10349]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Elasticsearch.

Nik1 · May 30, 2023, 12:08pm

Below are the messages shown when executed "journalctl -xe"

logstash[900]: [2023-05-30T12:05:01,163][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.
logstash[900]: [2023-05-30T12:05:06,167][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.
logstash[900]: [2023-05-30T12:05:11,172][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.
logstash[900]: [2023-05-30T12:05:16,176][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.
logstash[900]: [2023-05-30T12:05:21,180][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.
filebeat[917]: 2023-05-30T12:05:21.411Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elas
filebeat[917]: 2023-05-30T12:05:21.411Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff
filebeat[917]: 2023-05-30T12:05:21.411Z INFO [publisher] pipeline/retry.go:213 retryer: send wait signal to consumer
filebeat[917]: 2023-05-30T12:05:21.411Z INFO [publisher] pipeline/retry.go:217 done
logstash[900]: [2023-05-30T12:05:26,184][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.
filebeat[917]: 2023-05-30T12:05:30.592Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"
Mlogstash[900]: [2023-05-30T12:05:31,188][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.
logstash[900]: [2023-05-30T12:05:36,192][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.
logstash[900]: [2023-05-30T12:05:41,196][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.
logstash[900]: [2023-05-30T12:05:46,200][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error.

dadoonet · May 30, 2023, 12:57pm

Welcome!

Please read this about how to format.

You need to provide Elasticsearch logs. See /var/log/elasticsearch/elasticsearch.log as stated in the system logs you shared...

Nik1 · May 31, 2023, 10:26am

Got the below logs from `/var/log/elasticsearch/elasticsearch.log

[2023-05-31T10:04:00,919][ERROR][o.e.b.Bootstrap ] [elasticsearch] Exception
java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.2.jar:7.11.2]
Suppressed: java.lang.IllegalArgumentException: unknown secure setting [AKIA2OEL6HUQHLHZRQ57] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.2.jar:7.11.2]
[2023-05-31T10:04:00,927][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]
Caused by: java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
... 6 more
Suppressed: java.lang.IllegalArgumentException: unknown secure setting [AKIA2OEL6HUQHLHZRQ57] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]
[2023-05-31T10:12:25,324][INFO ][o.e.n.Node ] [elasticsearch] version[7.11.2], pid[17038], build[default/deb/3e5a16cfec50876d20ea77b075070932c6464c7d/2021-03-06T05:54:38.141101Z], OS[Linux/5.4.0-1057-aws/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2023-05-31T10:12:25,330][INFO ][o.e.n.Node ] [elasticsearch] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2023-05-31T10:12:25,332][INFO ][o.e.n.Node ] [elasticsearch] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms2g, -Xmx2g, -Des.allow_insecure_settings=true, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2023-05-31T10:12:29,707][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [aggs-matrix-stats]
[2023-05-31T10:12:29,707][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [analysis-common]
[2023-05-31T10:12:29,708][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [constant-keyword]
[2023-05-31T10:12:29,708][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [flattened]
[2023-05-31T10:12:29,708][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [frozen-indices]
[2023-05-31T10:12:29,709][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [ingest-common]
[2023-05-31T10:12:29,709][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [ingest-geoip]
[2023-05-31T10:12:29,709][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [ingest-user-agent]
[2023-05-31T10:12:29,710][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [kibana]
[2023-05-31T10:12:29,710][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [lang-expression]
[2023-05-31T10:12:29,711][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [lang-mustache]
[2023-05-31T10:12:29,711][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [lang-painless]
[2023-05-31T10:12:29,711][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [mapper-extras]
[2023-05-31T10:12:29,712][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [mapper-version]
[2023-05-31T10:12:29,712][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [parent-join]
[2023-05-31T10:12:29,712][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [percolator]
[2023-05-31T10:12:29,713][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [rank-eval]
[2023-05-31T10:12:29,714][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [reindex]
[2023-05-31T10:12:29,714][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [repositories-metering-api]
[2023-05-31T10:12:29,715][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [repository-url]
[2023-05-31T10:12:29,716][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [search-business-rules]
[2023-05-31T10:12:29,716][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [searchable-snapshots]
[2023-05-31T10:12:29,716][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [spatial]
[2023-05-31T10:12:29,717][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [systemd]
[2023-05-31T10:12:29,717][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [transform]
[2023-05-31T10:12:29,717][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [transport-netty4]
[2023-05-31T10:12:29,718][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [unsigned-long]
[2023-05-31T10:12:29,718][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [vectors]
[2023-05-31T10:12:29,718][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [wildcard]
[2023-05-31T10:12:29,718][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-aggregate-metric]
[2023-05-31T10:12:29,719][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-analytics]
[2023-05-31T10:12:29,719][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-async]
[2023-05-31T10:12:29,719][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-async-search]
[2023-05-31T10:12:29,720][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-autoscaling]
[2023-05-31T10:12:29,720][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-ccr]
[2023-05-31T10:12:29,720][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-core]
[2023-05-31T10:12:29,720][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-data-streams]
[2023-05-31T10:12:29,721][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-deprecation]
[2023-05-31T10:12:29,721][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-enrich]
[2023-05-31T10:12:29,721][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-eql]
[2023-05-31T10:12:29,722][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-fleet]
[2023-05-31T10:12:29,722][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-graph]
[2023-05-31T10:12:29,722][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-identity-provider]
[2023-05-31T10:12:29,723][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-ilm]
[2023-05-31T10:12:29,723][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-ingest]
[2023-05-31T10:12:29,723][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-logstash]
[2023-05-31T10:12:29,723][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-ml]
[2023-05-31T10:12:29,724][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-monitoring]
[2023-05-31T10:12:29,724][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-ql]
[2023-05-31T10:12:29,724][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-rollup]
[2023-05-31T10:12:29,724][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-runtime-fields]
[2023-05-31T10:12:29,725][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-security]
[2023-05-31T10:12:29,725][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-sql]
[2023-05-31T10:12:29,725][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-stack]
[2023-05-31T10:12:29,726][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-voting-only-node]
[2023-05-31T10:12:29,726][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded module [x-pack-watcher]
[2023-05-31T10:12:29,727][INFO ][o.e.p.PluginsService ] [elasticsearch] loaded plugin [repository-s3]
[2023-05-31T10:12:29,787][INFO ][o.e.e.NodeEnvironment ] [elasticsearch] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [205.7gb], net total_space [387.7gb], types [ext4]
[2023-05-31T10:12:29,788][INFO ][o.e.e.NodeEnvironment ] [elasticsearch] heap size [2gb], compressed ordinary object pointers [true]
[2023-05-31T10:12:31,712][INFO ][o.e.n.Node ] [elasticsearch] node name [elasticsearch], node ID [BVzpIFi0QcCX3pgcTFdZew], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-05-31T10:12:36,229][ERROR][o.e.b.Bootstrap ] [elasticsearch] Exception
java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.2.jar:7.11.2]
Suppressed: java.lang.IllegalArgumentException: unknown secure setting [AKIA2OEL6HUQHLHZRQ57] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.2.jar:7.11.2]
[2023-05-31T10:12:36,241][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]
Caused by: java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
... 6 more
Suppressed: java.lang.IllegalArgumentException: unknown secure setting [AKIA2OEL6HUQHLHZRQ57] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.node.Node.(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]

dadoonet · May 31, 2023, 12:13pm

Please don't post unformatted code, logs, or configuration as it's very hard to read.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

But anyway here, you defined a setting which does not exist. I suspect that you wanted to enter the value of the setting instead of its name.

Remove it (and regenerate your credentials by the way)

Nik1 · June 1, 2023, 11:32am

Oh pardon me, I missed to format the logs. I am posting the full log in two sections 

[2023-06-01T09:27:22,224][INFO ][o.e.n.Node               ] [elasticsearch] version[7.11.2], pid[937], build[default/deb/3e5a16cfec50876d20ea77b075070932c6464c7d/2021-03-06T05:54:38.141101Z], OS[Linux/5.4.0-1057-aws/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2023-06-01T09:27:22,519][INFO ][o.e.n.Node               ] [elasticsearch] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2023-06-01T09:27:22,662][INFO ][o.e.n.Node               ] [elasticsearch] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms2g, -Xmx2g, -Des.allow_insecure_settings=true, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2023-06-01T09:27:47,505][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [aggs-matrix-stats]
[2023-06-01T09:27:47,505][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [analysis-common]
[2023-06-01T09:27:47,506][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [constant-keyword]
[2023-06-01T09:27:47,506][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [flattened]
[2023-06-01T09:27:47,512][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [frozen-indices]
[2023-06-01T09:27:47,513][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [ingest-common]
[2023-06-01T09:27:47,513][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [ingest-geoip]
[2023-06-01T09:27:47,515][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [ingest-user-agent]
[2023-06-01T09:27:47,524][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [kibana]
[2023-06-01T09:27:47,525][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [lang-expression]
[2023-06-01T09:27:47,525][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [lang-mustache]
[2023-06-01T09:27:47,526][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [lang-painless]
[2023-06-01T09:27:47,526][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [mapper-extras]
[2023-06-01T09:27:47,528][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [mapper-version]
[2023-06-01T09:27:47,529][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [parent-join]
[2023-06-01T09:27:47,529][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [percolator]
[2023-06-01T09:27:47,530][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [rank-eval]
[2023-06-01T09:27:47,530][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [reindex]
[2023-06-01T09:27:47,540][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [repositories-metering-api]
[2023-06-01T09:27:47,541][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [repository-url]
[2023-06-01T09:27:47,541][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [search-business-rules]
[2023-06-01T09:27:47,541][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [searchable-snapshots]
[2023-06-01T09:27:47,543][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [spatial]
[2023-06-01T09:27:47,544][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [systemd]
[2023-06-01T09:27:47,544][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [transform]
[2023-06-01T09:27:47,545][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [transport-netty4]
[2023-06-01T09:27:47,545][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [unsigned-long]
[2023-06-01T09:27:47,546][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [vectors]
[2023-06-01T09:27:47,546][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [wildcard]
[2023-06-01T09:27:47,546][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-aggregate-metric]
[2023-06-01T09:27:47,547][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-analytics]
[2023-06-01T09:27:47,547][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-async]
[2023-06-01T09:27:47,547][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-async-search]
[2023-06-01T09:27:47,548][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-autoscaling]
[2023-06-01T09:27:47,548][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ccr]
[2023-06-01T09:27:47,556][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-core]
[2023-06-01T09:27:47,556][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-data-streams]
[2023-06-01T09:27:47,557][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-deprecation]
[2023-06-01T09:27:47,557][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-enrich]
[2023-06-01T09:27:47,558][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-eql]
[2023-06-01T09:27:47,559][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-fleet]
[2023-06-01T09:27:47,559][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-graph]
[2023-06-01T09:27:47,559][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-identity-provider]
[2023-06-01T09:27:47,560][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ilm]
[2023-06-01T09:27:47,569][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ingest]
[2023-06-01T09:27:47,569][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-logstash]
[2023-06-01T09:27:47,570][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ml]
[2023-06-01T09:27:47,570][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-monitoring]
[2023-06-01T09:27:47,570][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ql]
[2023-06-01T09:27:47,571][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-rollup]
[2023-06-01T09:27:47,572][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-runtime-fields]
[2023-06-01T09:27:47,572][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-security]
[2023-06-01T09:27:47,572][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-sql]
[2023-06-01T09:27:47,581][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-stack]
[2023-06-01T09:27:47,581][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-voting-only-node]
[2023-06-01T09:27:47,582][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-watcher]
[2023-06-01T09:27:47,586][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded plugin [repository-s3]
[2023-06-01T09:27:47,852][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [206.3gb], net total_space [387.7gb], types [ext4]
[2023-06-01T09:27:47,855][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch] heap size [2gb], compressed ordinary object pointers [true]
[2023-06-01T09:27:57,252][INFO ][o.e.n.Node               ] [elasticsearch] node name [elasticsearch], node ID [BVzpIFi0QcCX3pgcTFdZew], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-06-01T09:28:17,712][ERROR][o.e.b.Bootstrap          ] [elasticsearch] Exception
java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.2.jar:7.11.2]
        Suppressed: java.lang.IllegalArgumentException: unknown secure setting [AKIA2OEL6HUQHLHZRQ57] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.node.Node.<init>(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.2.jar:7.11.2]

Nik1 · June 1, 2023, 11:33am


[2023-06-01T09:28:17,745][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]
Caused by: java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
        ... 6 more
        Suppressed: java.lang.IllegalArgumentException: unknown secure setting [AKIA2OEL6HUQHLHZRQ57] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.node.Node.<init>(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]
[2023-06-01T09:44:17,184][INFO ][o.e.n.Node               ] [elasticsearch] version[7.11.2], pid[4288], build[default/deb/3e5a16cfec50876d20ea77b075070932c6464c7d/2021-03-06T05:54:38.141101Z], OS[Linux/5.4.0-1057-aws/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2023-06-01T09:44:17,190][INFO ][o.e.n.Node               ] [elasticsearch] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2023-06-01T09:44:17,191][INFO ][o.e.n.Node               ] [elasticsearch] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms2g, -Xmx2g, -Des.allow_insecure_settings=true, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2023-06-01T09:44:22,579][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [aggs-matrix-stats]
[2023-06-01T09:44:22,580][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [analysis-common]
[2023-06-01T09:44:22,581][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [constant-keyword]
[2023-06-01T09:44:22,581][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [flattened]
[2023-06-01T09:44:22,582][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [frozen-indices]
[2023-06-01T09:44:22,583][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [ingest-common]
[2023-06-01T09:44:22,583][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [ingest-geoip]
[2023-06-01T09:44:22,583][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [ingest-user-agent]
[2023-06-01T09:44:22,584][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [kibana]
[2023-06-01T09:44:22,584][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [lang-expression]
[2023-06-01T09:44:22,584][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [lang-mustache]
[2023-06-01T09:44:22,585][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [lang-painless]
[2023-06-01T09:44:22,585][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [mapper-extras]
[2023-06-01T09:44:22,586][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [mapper-version]
[2023-06-01T09:44:22,586][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [parent-join]
[2023-06-01T09:44:22,587][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [percolator]
[2023-06-01T09:44:22,587][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [rank-eval]
[2023-06-01T09:44:22,588][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [reindex]
[2023-06-01T09:44:22,588][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [repositories-metering-api]
[2023-06-01T09:44:22,589][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [repository-url]
[2023-06-01T09:44:22,589][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [search-business-rules]
[2023-06-01T09:44:22,590][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [searchable-snapshots]
[2023-06-01T09:44:22,591][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [spatial]
[2023-06-01T09:44:22,592][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [systemd]
[2023-06-01T09:44:22,592][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [transform]
[2023-06-01T09:44:22,593][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [transport-netty4]
[2023-06-01T09:44:22,594][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [unsigned-long]
[2023-06-01T09:44:22,594][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [vectors]
[2023-06-01T09:44:22,595][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [wildcard]
[2023-06-01T09:44:22,595][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-aggregate-metric]
[2023-06-01T09:44:22,595][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-analytics]
[2023-06-01T09:44:22,596][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-async]
[2023-06-01T09:44:22,597][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-async-search]
[2023-06-01T09:44:22,597][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-autoscaling]
[2023-06-01T09:44:22,598][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ccr]
[2023-06-01T09:44:22,598][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-core]
[2023-06-01T09:44:22,598][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-data-streams]
[2023-06-01T09:44:22,599][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-deprecation]
[2023-06-01T09:44:22,599][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-enrich]
[2023-06-01T09:44:22,600][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-eql]
[2023-06-01T09:44:22,603][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-fleet]
[2023-06-01T09:44:22,603][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-graph]
[2023-06-01T09:44:22,604][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-identity-provider]
[2023-06-01T09:44:22,604][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ilm]
[2023-06-01T09:44:22,605][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ingest]
[2023-06-01T09:44:22,605][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-logstash]
[2023-06-01T09:44:22,605][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ml]
[2023-06-01T09:44:22,605][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-monitoring]
[2023-06-01T09:44:22,606][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-ql]
[2023-06-01T09:44:22,607][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-rollup]
[2023-06-01T09:44:22,607][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-runtime-fields]
[2023-06-01T09:44:22,607][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-security]
[2023-06-01T09:44:22,608][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-sql]
[2023-06-01T09:44:22,608][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-stack]
[2023-06-01T09:44:22,609][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-voting-only-node]
[2023-06-01T09:44:22,610][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded module [x-pack-watcher]
[2023-06-01T09:44:22,614][INFO ][o.e.p.PluginsService     ] [elasticsearch] loaded plugin [repository-s3]
[2023-06-01T09:44:22,736][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [206.2gb], net total_space [387.7gb], types [ext4]
[2023-06-01T09:44:22,739][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch] heap size [2gb], compressed ordinary object pointers [true]
[2023-06-01T09:44:24,807][INFO ][o.e.n.Node               ] [elasticsearch] node name [elasticsearch], node ID [BVzpIFi0QcCX3pgcTFdZew], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2023-06-01T09:44:29,672][ERROR][o.e.b.Bootstrap          ] [elasticsearch] Exception
java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.2.jar:7.11.2]
        Suppressed: java.lang.IllegalArgumentException: unknown secure setting [AKIA2OEL6HUQHLHZRQ57] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.node.Node.<init>(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.2.jar:7.11.2]
[2023-06-01T09:44:29,683][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]
Caused by: java.lang.IllegalArgumentException: unknown setting [allow_insecure_settings] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
        ... 6 more
        Suppressed: java.lang.IllegalArgumentException: unknown secure setting [AKIA2OEL6HUQHLHZRQ57] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.node.Node.<init>(Node.java:395) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:216) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:387) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
                at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]

Christian_Dahlqvist · June 1, 2023, 11:45am

Please share your elasticsearch.yml config file.

Nik1 · June 1, 2023, 12:24pm

The "elasticsearch.yml" config is as below,

network.host: 127.0.0.1
node.name: elasticsearch
cluster.initial_master_nodes: elasticsearch

# Transport layer
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

# HTTP layer
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

# Elasticsearch authentication
xpack.security.enabled: true
allow_insecure_settings: true

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

Christian_Dahlqvist · June 1, 2023, 12:41pm

This is according to the logs not a valid settings. Remove it and try starting the node.

Nik1 · June 1, 2023, 1:01pm

allow_insecure_settings: true >> This was added later for troubleshooting purpose. I removed it now and tried starting elasticsearch service. The result is same

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2023-06-01 12:58:06 UTC; 1min 24s ago
     Docs: https://www.elastic.co
  Process: 7754 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 7754 (code=exited, status=1/FAILURE)

systemd-entrypoint[7754]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
systemd-entrypoint[7754]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
systemd-entrypoint[7754]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
systemd-entrypoint[7754]:         at org.elasticsearch.cli.Command.main(Command.java:79)
systemd-entrypoint[7754]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
systemd-entrypoint[7754]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
systemd-entrypoint[7754]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Elasticsearch.

Christian_Dahlqvist · June 1, 2023, 1:04pm

Please show the Elasticsearch logs and the error(s) available there. I doubt the error message will be exactly the same.

leandrojmp · June 1, 2023, 1:16pm

Also look at the system logs on /var/log/messages or /var/log/syslog, this will tell you why the systemd service can not start.

Sometimes the service fails before anything can be logged in Elasticsearch logs, so you will only find the reason on the system logs.

dadoonet · June 1, 2023, 1:28pm

Yeah. It's often caused by a bad indent in the config file (which @Nik1 just edited )

Nik1 · June 1, 2023, 1:51pm

@Christian_Dahlqvist The logs from `/var/log/elasticsearch/elasticsearch.log" is same as before I posted, no changes

leandrojmp · June 1, 2023, 2:16pm

You need to check the systems logs.

dadoonet · June 1, 2023, 4:56pm

If no changes, it means that you still have this line in the configuration:

allow_insecure_settings: true

Nik1 · June 2, 2023, 9:53am

@dadoonet No, it not there now. "allow_insecure_settings: true" is removed from elasticsearch.yml config file.

network.host: 127.0.0.1
node.name: elasticsearch
cluster.initial_master_nodes: elasticsearch

# Transport layer
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

# HTTP layer
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

# Elasticsearch authentication
xpack.security.enabled: true

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
root@ip-10-0-50-220:/etc/elasticsearch# cat elasticsearch.yml
network.host: 127.0.0.1
node.name: elasticsearch
cluster.initial_master_nodes: elasticsearch

# Transport layer
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.transport.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

# HTTP layer
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt

# Elasticsearch authentication
xpack.security.enabled: true

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
root@ip-10-0-50-220:/etc/elasticsearch#

Nik1 · June 2, 2023, 10:05am

Below is the "/var/log/syslog" displayed after the Elasticsearch service start failure

filebeat[950]: 2023-06-02T09:58:18.112Z#011INFO#011[publisher]#011pipeline/retry.go:217#011  done
filebeat[950]: 2023-06-02T09:58:18.112Z#011INFO#011[publisher_pipeline_output]#011pipeline/output.go:145#011Attempting to reconnect to backoff(elasticsearch(https://127.0.0.1:9200)) with 1966 reconnect attempt(s)
filebeat[950]: 2023-06-02T09:58:19.552Z#011INFO#011[monitoring]#011log/log.go:144#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":5907899}},"memory":{"mem":{"usage":{"bytes":-450560}}}},"cpu":{"system":{"ticks":12970,"time":{"ms":7}},"total":{"ticks":40100,"time":{"ms":7},"value":40100},"user":{"ticks":27130}},"handles":{"limit":{"hard":4096,"soft":1024},"open":12},"info":{"ephemeral_id":"72a6c7d0-0f90-41e8-ba3c-ae1e5055afcf","uptime":{"ms":88291179}},"memstats":{"gc_next":56906016,"memory_alloc":29138360,"memory_total":1523845512,"rss":54398976},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":2,"running":1}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":4117,"retry":50}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":2.62,"15":0.76,"5":1.65,"norm":{"1":1.31,"15":0.38,"5":0.825}}}}}}
logstash[899]: [2023-06-02T09:58:22,539][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash[899]: [2023-06-02T09:58:27,542][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash[899]: [2023-06-02T09:58:32,544][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash[899]: [2023-06-02T09:58:37,547][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash[899]: [2023-06-02T09:58:42,551][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash[899]: [2023-06-02T09:58:47,555][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@127.0.0.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
filebeat[950]: 2023-06-02T09:58:49.552Z#011INFO#011[monitoring]#011log/log.go:144#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":6567330}},"memory":{"mem":{"usage":{"bytes":-1323008}}}},"cpu":{"system":{"ticks":12980,"time":{"ms":3}},"total":{"ticks":40110,"time":{"ms":3},"value":40110},"user":{"ticks":27130}},"handles":{"limit":{"hard":4096,"soft":1024},"open":12},"info":{"ephemeral_id":"72a6c7d0-0f90-41e8-ba3c-ae1e5055afcf","uptime":{"ms":88321178}},"memstats":{"gc_next":56906016,"memory_alloc":29364712,"memory_total":1524071864,"rss":52944896},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":2,"running":1}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":4117}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":2.17,"15":0.78,"5":1.64,"norm":{"1":1.085,"15":0.39,"5":0.82}}}}}}

dadoonet · June 2, 2023, 10:41am

But you said logs are the same.

If it's right, logs are still saying something like:

unknown setting [allow_insecure_settings]

If not, please provide Elasticsearch logs.