Elasticsearch 7.13

`rypoint[1835]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
rypoint[1835]: output:
rypoint[1835]: [0.029s][error][logging] Error opening log file '/var/log/elasticsearch/gc.log': No space left on device
rypoint[1835]: [0.030s][error][logging] Initialization of output 'file=/var/log/elasticsearch/gc.log' using options 'fil
rypoint[1835]: error:
rypoint[1835]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tag
rypoint[1835]: Error: Could not create the Java Virtual Machine.
rypoint[1835]: Error: A fatal exception has occurred. Program will exit.
rypoint[1835]: at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
rypoint[1835]: at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:81)
rypoint[1835]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38)
rypoint[1835]: at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:135)
rypoint[1835]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
 elasticsearch.service: main process exited, code=exited, status=1/FAILURE
 Failed to start Elasticsearch.
 Unit elasticsearch.service entered failed state.`

After restart of ES, it create an error.

It's not clear what you are asking or looking for by just posting part of a log entry.
Please add more information to allow us to help you.

1 Like

Hello sir ,
I am not able to start elasticsearch.

This is the log file

[2021-06-09T11:03:05,369][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [terrena-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - truststore file [/etc/elasticsearch/elastic-certificates.p12] does not exist]; nested: NoSuchFileException[/etc/elasticsearch/elastic-certificates.p12];
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.13.0.jar:7.13.0]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:454) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:298) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$18(Node.java:605) ~[elasticsearch-7.13.0.jar:7.13.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:609) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.13.0.jar:7.13.0]
        ... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager - truststore file [/etc/elasticsearch/elastic-certificates.p12] does not exist
        at org.elasticsearch.xpack.core.ssl.TrustConfig.missingTrustConfigFile(TrustConfig.java:114) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:69) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1224) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:454) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:298) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$18(Node.java:605) ~[elasticsearch-7.13.0.jar:7.13.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:609) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.13.0.jar:7.13.0]
        ... 6 more
Caused by: java.nio.file.NoSuchFileException: /etc/elasticsearch/elastic-certificates.p12
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:375) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:426) ~[?:?]
        at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
        at java.nio.file.Files.newInputStream(Files.java:160) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:96) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:66) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:439) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1224) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:454) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:298) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$18(Node.java:605) ~[elasticsearch-7.13.0.jar:7.13.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:609) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.13.0.jar:7.13.0]
        ... 6 more

Ok, but what are you trying to do? What is not happening?

I am not able to start elasticsearch..
I would like to setup basic security which is required to create an alert... while doing that work i have done some changes in the elasticsearch.yml and I have restarted ES... but unfortunately, it is not working

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: terrena
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: terrena-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
node.master: true
node.data: true
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0

# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["10.xxx.xxx.141"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
action.destructive_requires_name: true
xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

That seems relevant.

1 Like

Thanks ..... Yes i have copied the certificate but then also it shows an error

`[2021-06-09T12:42:07,155][ERROR][o.e.b.Bootstrap          ] [terrena-1] Exception
java.lang.IllegalArgumentException: unknown secure setting [xpack.security.transport.ssl] did you mean any of [xpack.security.transport.ssl.key, xpack.security.transport.ssl.enabled]?
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:437) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.13.0.jar:7.13.0]
[2021-06-09T12:42:07,159][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [terrena-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [xpack.security.transport.ssl] did you mean any of [xpack.security.transport.ssl.key, xpack.security.transport.ssl.enabled]?
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.13.0.jar:7.13.0]
Caused by: java.lang.IllegalArgumentException: unknown secure setting [xpack.security.transport.ssl] did you mean any of [xpack.security.transport.ssl.key, xpack.security.transport.ssl.enabled]?
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:437) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.13.0.jar:7.13.0]
        ... 6 more`

You create a setting named xpack.security.transport.ssl with elasticsearch-keystore, but that is not a valid setting name.

You should remove it with elasticsearch-keystore remove

Thanks @TimV
I have removed elasticsearch-keystore then also error remains same..

[2021-06-09T14:18:50,940][ERROR][o.e.b.Bootstrap          ] [terrena-1] Exception
java.lang.IllegalArgumentException: unknown secure setting [xpack.security.transport.ssl] did you mean any of [xpack.security.transport.ssl.key, xpack.security.transport.ssl.enabled]?
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:437) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.13.0.jar:7.13.0]
[2021-06-09T14:18:50,943][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [terrena-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [xpack.security.transport.ssl] did you mean any of [xpack.security.transport.ssl.key, xpack.security.transport.ssl.enabled]?
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.13.0.jar:7.13.0]
Caused by: java.lang.IllegalArgumentException: unknown secure setting [xpack.security.transport.ssl] did you mean any of [xpack.security.transport.ssl.key, xpack.security.transport.ssl.enabled]?
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:437) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.13.0.jar:7.13.0]
        ... 6 more

elasticsearch.yml

#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["10.xx.xx.141"]
discovery.type: single-node
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

What exactly did you do?

Just i want to implement xpack in elasticsearch, so that i can able to create an alert but when i followed the steps which are mention here

but getting error after restart of the elasticsearch....

I would focus on this message from your log excerpt:

Error opening log file '/var/log/elasticsearch/gc.log': No space left on device
sudo /usr/share/elasticsearch/bin/elasticsearch-keystore remove
A tool for managing settings stored in the elasticsearch keystore

Non-option arguments:
command

Option             Description
------             -----------
-E <KeyValuePair>  Configure a setting
-h, --help         Show help
-s, --silent       Show minimal output
-v, --verbose      Show verbose output
ERROR: Must supply at least one setting to remove

I have removed the setting named xpack.security.transport.ssl with elastic-keystore....

[2021-06-10T10:48:05,749][ERROR][o.e.b.Bootstrap          ] [terrena-1] Exception
java.lang.IllegalArgumentException: unknown secure setting [remove] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:420) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:138) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:437) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:217) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:397) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.13.0.jar:7.13.0]
[2021-06-10T10:48:05,761][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [terrena-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown secure setting [remove] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.13.0.jar:7.13.0]
Caused by: java.lang.IllegalArgumentException: unknown secure setting [remove] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:533) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:478) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:449) ~[elasticsearch-7.13.0.jar:7.13.0]

You added in the elasticsearch.yml file a setting named remove. This is not supported.

If you don't know what you did, please provide the last version of your elasticsearch.yml file.

1 Like

@Krunal, please let me know if any of the documentation seems unclear, or if there were parts where you got stuck.

I have configured the basic security as mentioned on this link...

`

elasticsearch.yml
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: terrena
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: terrena-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
node.master: true
node.data: true
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0

# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.type: single-node
discovery.seed_hosts: ["10.xxx.xxx.xxx"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
[2021-06-22T16:27:18,704][ERROR][o.e.b.Bootstrap          ] [terrena-1] Exception
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]

[2021-06-22T16:27:18,712][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [terrena-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.transport.ssl]]; nested: ElasticsearchException[failed to initialize SSL TrustManager - not permitted to read truststore file [/etc/elasticsearch/elastic-certificates.p12]]; nested: AccessDeniedException[/etc/elasticsearch/elastic-certificates.p12];
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.13.0.jar:7.13.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.13.0.jar:7.13.0]


It seems that your keystore file is in /etc/elasticsearch/elastic-certificates.p12. Can you move the elastic-certificates.p12 file to the Elasticsearch root directory and then try to restart Elasticsearch? Also, it looks like xpack.security.transport.ssl.truststore.path is missing from your elasticsearch.yml file. Perhaps try this configuration?

xpack.security.transport.ssl.enabled: true
xpack.security.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12