Elasticsearch stop working after enable x-pack

ikakavas · August 13, 2019, 5:39pm

Did you do this:

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the --quiet option must be removed from the ExecStart command line in the elasticsearch.service file.
When systemd logging is enabled, the logging information are available using the journalctl commands:

From the documentation link I shared with you ?

Ayush_Verma · August 13, 2019, 6:24pm

I have been trying this can you please share a link that demonstrate how to do that ??

journalctl -u elasticsearch.service

Aug 13 18:07:54 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 18:07:57 elastic-stage-vyakar elasticsearch[3897]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Aug 13 18:08:21 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Aug 13 18:08:21 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Aug 13 18:18:13 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 18:18:15 elastic-stage-vyakar elasticsearch[4164]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Aug 13 18:18:40 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Aug 13 18:18:40 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

is this help full.

ikakavas · August 14, 2019, 5:45am

The instructions are in the text I have copied above.

I'm not sure what is not clear in the above Just get your elasticsearch.service file, which is located in /usr/lib/systemd/system

open it with the editor of your choice, find the line that says

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

and change it to

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid

finally run

sudo systemctl daemon-reload

to reload the service file.

Now attempt to start Elasticsearch and if it fails, the logs will be available when you run

sudo journalctl --unit elasticsearch

Ayush_Verma · August 14, 2019, 5:57am

Now i understand, This step by step process i can understand, you are really a saviour now i guess logs are getting generated by journalctl -u elasticsearch.service

Aug 14 05:49:32 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:32,250][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
Aug 14 05:49:33 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:33,972][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/2075] [Main.cc@110] controller (64 bit): Version 7.3.0 (Build ff2f774f78ce63) Copyright (c) 2019 Elasticsearch BV
Aug 14 05:49:34 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:34,853][DEBUG][o.e.a.ActionModule       ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
Aug 14 05:49:35 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:35,647][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and seed hosts providers [settings]
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:37,420][INFO ][o.e.n.Node               ] [node-1] initialized
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:37,427][INFO ][o.e.n.Node               ] [node-1] starting ...
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:37,775][INFO ][o.e.t.TransportService   ] [node-1] publish_address {10.128.0.26:9300}, bound_addresses {10.128.0.26:9300}
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:37,798][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: ERROR: [1] bootstrap checks failed
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:37,882][INFO ][o.e.n.Node               ] [node-1] stopping ...
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:37,912][INFO ][o.e.n.Node               ] [node-1] stopped
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:37,915][INFO ][o.e.n.Node               ] [node-1] closing ...
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:37,947][INFO ][o.e.n.Node               ] [node-1] closed
Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [2019-08-14T05:49:37,951][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started
Aug 14 05:49:38 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Aug 14 05:49:38 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

i really don't how should i thank you atleast logs are getting generated.

ikakavas · August 14, 2019, 6:02am

Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: ERROR: [1] bootstrap checks failed Aug 14 05:49:37 elastic-stage-vyakar elasticsearch[1986]: [1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]

This is self-explanatory: You need to enable TLS on the transport layer when you enable security. In the Documentation that I have shared with you a couple of times before https://www.elastic.co/guide/en/elasticsearch/reference/7.3/configuring-security.html, this is step 4, which is before step 5 that says "Start Elasticsearch" . So you need to configure TLS for transport and then start Elastisearch.

Ayush_Verma · August 14, 2019, 6:25am

Yes now i am getting, so according to link Configure Transport Layer Security (TLS/SSL) for internode-communication.
this is what i have to do next

and i am trying to generate private key by following this step, 2. Generate a private key and X.509 certificate (1st step is enbale xpack.security.enabled that i did).

i used this command bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
then its asked me to enter password for and this is what i get after that.

Enter password for CA (elastic-stack-ca.p12) : 
Exception in thread "main" java.nio.file.NoSuchFileException: elastic-stack-ca.p12
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:373)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:424)
        at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
        at java.base/java.nio.file.Files.newInputStream(Files.java:158)
        at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readKeyStore(CertParsingUtils.java:68)
        at org.elasticsearch.xpack.core.ssl.CertParsingUtils.readPkcs12KeyPairs(CertParsingUtils.java:130)
        at org.elasticsearch.xpack.security.cli.CertificateTool$CertificateCommand.lambda$loadPkcs12CA$1(CertificateTool.java:341)
        at org.elasticsearch.xpack.security.cli.CertificateTool.withPassword(CertificateTool.java:931)
        at org.elasticsearch.xpack.security.cli.CertificateTool.access$100(CertificateTool.java:85)
        at org.elasticsearch.xpack.security.cli.CertificateTool$CertificateCommand.loadPkcs12CA(CertificateTool.java:340)
        at org.elasticsearch.xpack.security.cli.CertificateTool$CertificateCommand.getCAInfo(CertificateTool.java:328)
        at org.elasticsearch.xpack.security.cli.CertificateTool$GenerateCertificateCommand.execute(CertificateTool.java:684)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.xpack.security.cli.CertificateTool.main(CertificateTool.java:137)

Ayush_Verma · August 14, 2019, 6:31am

sorry this is what i get now

The 'cert' mode generates X.509 certificate and private keys.
    * By default, this generates a single certificate and key for use
       on a single instance.
    * The '-multiple' option will prompt you to enter details for multiple
       instances and will generate a certificate and key for each one
    * The '-in' option allows for the certificate generation to be automated by describing
       the details of each instance in a YAML file
    * An instance is any piece of the Elastic Stack that requires a SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.
    * All certificates generated by this tool will be signed by a certificate authority (CA).
    * The tool can automatically generate a new CA for you, or you can provide your own with the
         -ca or -ca-cert command line options.
By default the 'cert' mode produces a single PKCS#12 output file which holds:
    * The instance certificate
    * The private key for the instance certificate
    * The CA certificate
If you specify any of the following options:
    * -pem (PEM formatted output)
    * -keep-ca-key (retain generated CA key)
    * -multiple (generate multiple certificates)

ikakavas · August 14, 2019, 6:39am

We're spending significant time in this thread, I will urge you to please try and read through all the instructions before posting. You are missing step 1, please go through this one more time and execute all the commands in order.

Ayush_Verma · August 14, 2019, 8:06am

Now

Verify that the xpack.security.enabled setting is true (i did that).
Generate a private key and X.509 Certificate (i did that by using following commands :- )
(a) bin/elasticsearch-certutil cert then this comes :- Please enter the desired output file [elastic-certificates.p12]: elastic-certificates.p12
for password, i leave it blank by pressing enter. It created file
Certificates written to /usr/share/elasticsearch/elastic-certificates.p12
i copy this file to /etc/elasticsearch/
Configure each node to: ( i did that )
by copy paste this into yml file

xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

Then i restart elasticsearch service by service elasticseaerch restart
and i got error.

this is my elasticsearch.yml file.

# Use a descriptive name for your cluster:
#
cluster.name: ElasticsearchStaging
#xpack.security.enabled: true
#xpack.security.audit.enabled: true
#xpack.security.http.ssl.enabled: true
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
network.host: 10.128.*.**
#
path.logs: /var/log/elasticsearch
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: 10.128.*.**
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: "node-1"
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

and these are logs:

Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:434) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:146) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.0.jar:7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.0.jar:7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.0.jar:7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.node.Node.<init>(Node.java:314) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.0.jar:7.3.0]

Ayush_Verma · August 14, 2019, 8:07am

even after following the steps i am getting error this thing is really irritating me. Please help me out.

ikakavas · August 14, 2019, 8:23am

These are not all the logs , these are just a few lines. We can't see what is going on if you don't share everything with us. If you're not sure how to see all logs from journalctl, try

journalctl -u elasticsearch.service --no-pager > es.log

then open es.log and share all the relevant parts.

There many things one can do wrong while following steps, so there is no way to magically know what's wrong unless you share all the information. I understand the irritation part, but do note that people in the community have been trying to help you promptly and with detailed answers through all your attempts, so you also need to remain patient if you want to get help to solve your issues.

antcas · August 14, 2019, 8:28am

@Ayush_Verma I will post what we did in our case to make everything work in our PROD cluster. Please bear in mind that I didn't read the whole topic with 100% focus and that my post might not be the solution for your problem.

Anyway, here it goes:

So, first of all we created the pair ca.key/ca.cert using the elasticsearch-certutil tool, with ca as an option.
Then we created the one p12 certificate in each node, using the key/cert pair mentioned before: elasticsearch-certutil cert --ca-cert YOUR/PATH/ca.crt --ca-key YOUR/PATH/ca.key --out elastic-certificates.p12 --pass ""
Please note that you need to use the same key/cert pair in all the nodes in order to be able to create valid certificates, with which nodes can talk to eachothers
After this, we changed the configuration files (like it is mentioned in the articles already posted here).
We stopped all the nodes of the cluster and then we started them again, one by one.
Authentication was enabled successfully and all the nodes could talk to eachother.

Hope this helps

Ayush_Verma · August 14, 2019, 8:36am

yes i will remain patient and i also know you guys are trying very hard and i really really appreciate that, this is what i get.

tail -100 es.log

Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,383][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [1015.6mb], compressed ordinary object
 pointers [true]
Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,421][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [eytGM4YaSh6r2vvbXRO
bxQ], cluster name [ElasticsearchStaging]
Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,423][INFO ][o.e.n.Node               ] [node-1] version[7.3.0], pid[4748], build[default/deb/de7
77fa/2019-07-24T18:30:11.767338Z], OS[Linux/4.15.0-1037-gcp/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,424][INFO ][o.e.n.Node               ] [node-1] JVM home [/usr/share/elasticsearch/jdk]
Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,427][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkS
weepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreT
ouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true
, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-17526956130986095316, -XX:+Heap
DumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elastics
earch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=
/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:57,593][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [
main]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elast
icsearch.xpack.core.XPackPlugin]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3
.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.0.jar:7.3
.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]: Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.3.0.jar:7.3
.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.0.jar:7.3
.0]

ikakavas · August 14, 2019, 8:46am

This is still not all the logs, you just cut it where the interesting part starts. Please, let us try to minimize the back and forth of communications by attempting to follow instructions to the point.

Share the whole file, or at least the full stack trace.

Ayush_Verma · August 14, 2019, 9:09am

Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:30
5) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:
139) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) 
[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsea
rch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7
.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch
-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java
:73) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:
59) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTra
nsportService.java:441) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportServic
e.java:1111) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:
1223) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1197
) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch
-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticse
arch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70) [elasticsearch-7.3.0.jar:7.3.0
]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:64) [elasticsearch-7.3.0.jar:7.3.
0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3
.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.0.j
ar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadConte
xt.java:758) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3
.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]

Ayush_Verma · August 14, 2019, 9:15am

there are lots of log i can't paste the whole thing is there any other way to share ??

ikakavas · August 14, 2019, 9:19am

Aug 14 07:28:46 elastic-stage-vyakar elasticsearch[4538]: Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12

As the error itself says, elasticsearch can't read the elastic-certificates.p12 file .

This is again already covered in the documentation that you have read, in the section https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html#deb-configuring

You need to make sure elasticsearch can read the file, do the following:

sudo chown root:elasticsearch /etc/elasticsearch/elastic-certificates.p12

Ayush_Verma · August 14, 2019, 10:13am

Hey Thanks now elasticsearch is not getting crashed after doing this sudo chown root:elasticsearch /etc/elasticsearch/elastic-certificates.p12 and also i added chmod 777 /etc/elasticsearch/elastic-certificates.p12 as u mentioned . But this is what i am getting after

curl -X GET '10.128.*.**:9200/'

elastic-stage-vyakar:/etc/elasticsearch# curl -X GET '10.128.0.26:9200/'
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" 
charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" cha
rset=\"UTF-8\""}},"status":401}

i also tried this.

curl -u elastic:changeme http://10.128.0.26:9200/

{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""
}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

But status of elasticsearch is active.

this is log :

Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]: Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.nio.file.Files.newByteChannel(Files.java:373) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.nio.file.Files.newByteChannel(Files.java:424) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.nio.file.Files.newInputStream(Files.java:158) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:434) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:146) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:
?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
45) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.0.jar:7.3
.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.0.jar:7.3
.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.0.jar:7.
3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.node.Node.<init>(Node.java:314) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         ... 6 more
Aug 14 07:30:57 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Aug 14 07:30:57 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

ikakavas · August 14, 2019, 10:53am

Please, please, please read the documentation. Almost all your questions in this post could be solved by reading the docs and following instructions instead of simply trying things and posting your errors.

You need to configure Elasticsearch. There is no user with changeme password so of course you cannot authenticate as such.

The doc we have been mentioning all this time ends with a link to

Learn how to configure Elasticsearch. There you can see Configure security in Elasticsearch | Elasticsearch Guide [8.11] | Elastic. Follow the docs.

This is just 10 lines from the log from 2 hours ago, before you solved the problem you had with the file and as such are not useful now.

You need to start putting some effort in solving your problems on your side too, otherwise I'm sorry but I'll have to disengage as this is becoming very time consuming and is unfair for the rest of the users of this forum.

Ayush_Verma · August 15, 2019, 3:57am

I am sorry that i am getting it very slow, i set password for elastic and kibana as well, but can you please tell me how to choose native realm i made a changes in elasticsearch.yml file here it is :

#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.audit.enabled: true
#xpack.monitoring.enabled: true
xpack.security.authc.realms.native: xpack:
  security:
    authc:
      realms:
        native:
          native1:
            order: 0
#---------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

can you please tell what is the right way to configure native realm.

kibana.yml:

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: kibanaindx

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"

Topic		Replies	Views
Elastic Search can't setup password for Kibana Elasticsearch elastic-stack-security	14	4685	September 11, 2019
Unable to login in Kibana after enable x-pack Kibana elastic-stack-security	10	7417	August 13, 2021
Inability to set up Shield/Security within X-Pack Trial Elasticsearch elastic-stack-security	7	721	August 31, 2018
Elasticsearch doesn't start anymore after x-pack settings Elasticsearch	5	5033	March 12, 2018
How to disable es security but enable kibana security Kibana	2	450	January 8, 2018

Elasticsearch stop working after enable x-pack

Related topics