Elasticsearch stop working after enable x-pack

Now

  1. Verify that the xpack.security.enabled setting is true (i did that).

  2. Generate a private key and X.509 Certificate (i did that by using following commands :- )
    (a) bin/elasticsearch-certutil cert then this comes :- Please enter the desired output file [elastic-certificates.p12]: elastic-certificates.p12
    for password, i leave it blank by pressing enter. It created file
    Certificates written to /usr/share/elasticsearch/elastic-certificates.p12
    i copy this file to /etc/elasticsearch/

  3. Configure each node to: ( i did that )
    by copy paste this into yml file

xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

  1. Then i restart elasticsearch service by service elasticseaerch restart
    and i got error.

this is my elasticsearch.yml file.

# Use a descriptive name for your cluster:
#
cluster.name: ElasticsearchStaging
#xpack.security.enabled: true
#xpack.security.audit.enabled: true
#xpack.security.http.ssl.enabled: true
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
network.host: 10.128.*.**
#
path.logs: /var/log/elasticsearch
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: 10.128.*.**
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: "node-1"
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

and these are logs:

Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:434) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:146) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.0.jar:7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.0.jar:7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.0.jar:7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.node.Node.<init>(Node.java:314) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.0.jar:7.3.0]

even after following the steps i am getting error this thing is really irritating me. Please help me out.

These are not all the logs , these are just a few lines. We can't see what is going on if you don't share everything with us. If you're not sure how to see all logs from journalctl, try

journalctl -u elasticsearch.service --no-pager > es.log 

then open es.log and share all the relevant parts.

There many things one can do wrong while following steps, so there is no way to magically know what's wrong unless you share all the information. I understand the irritation part, but do note that people in the community have been trying to help you promptly and with detailed answers through all your attempts, so you also need to remain patient if you want to get help to solve your issues.

1 Like

@Ayush_Verma I will post what we did in our case to make everything work in our PROD cluster. Please bear in mind that I didn't read the whole topic with 100% focus and that my post might not be the solution for your problem.

Anyway, here it goes:

  1. So, first of all we created the pair ca.key/ca.cert using the elasticsearch-certutil tool, with ca as an option.
  2. Then we created the one p12 certificate in each node, using the key/cert pair mentioned before: elasticsearch-certutil cert --ca-cert YOUR/PATH/ca.crt --ca-key YOUR/PATH/ca.key --out elastic-certificates.p12 --pass ""
    Please note that you need to use the same key/cert pair in all the nodes in order to be able to create valid certificates, with which nodes can talk to eachothers
  3. After this, we changed the configuration files (like it is mentioned in the articles already posted here).
  4. We stopped all the nodes of the cluster and then we started them again, one by one.
  5. Authentication was enabled successfully and all the nodes could talk to eachother.

Hope this helps

1 Like

yes i will remain patient and i also know you guys are trying very hard and i really really appreciate that, this is what i get.

tail -100 es.log

Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,383][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [1015.6mb], compressed ordinary object
 pointers [true]
Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,421][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [eytGM4YaSh6r2vvbXRO
bxQ], cluster name [ElasticsearchStaging]
Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,423][INFO ][o.e.n.Node               ] [node-1] version[7.3.0], pid[4748], build[default/deb/de7
77fa/2019-07-24T18:30:11.767338Z], OS[Linux/4.15.0-1037-gcp/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,424][INFO ][o.e.n.Node               ] [node-1] JVM home [/usr/share/elasticsearch/jdk]
Aug 14 07:30:56 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:56,427][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkS
weepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreT
ouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true
, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-17526956130986095316, -XX:+Heap
DumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elastics
earch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=
/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]: [2019-08-14T07:30:57,593][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [
main]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]: org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elast
icsearch.xpack.core.XPackPlugin]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3
.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.0.jar:7.3
.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]: Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.3.0.jar:7.3
.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.0.jar:7.3
.0]

This is still not all the logs, you just cut it where the interesting part starts. Please, let us try to minimize the back and forth of communications by attempting to follow instructions to the point.

Share the whole file, or at least the full stack trace.

Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:30
5) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:
139) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:264) 
[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:105) [elasticsea
rch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.InitialSearchPhase.access$200(InitialSearchPhase.java:50) [elasticsearch-7
.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:273) [elasticsearch
-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java
:73) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:
59) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTra
nsportService.java:441) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportServic
e.java:1111) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:
1223) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1197
) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch
-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticse
arch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:70) [elasticsearch-7.3.0.jar:7.3.0
]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:64) [elasticsearch-7.3.0.jar:7.3.
0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3
.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.3.0.j
ar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadConte
xt.java:758) [elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3
.0.jar:7.3.0]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
Aug 14 07:22:15 elastic-stage-vyakar elasticsearch[4089]:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]

there are lots of log i can't paste the whole thing is there any other way to share ??

Aug 14 07:28:46 elastic-stage-vyakar elasticsearch[4538]: Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12

As the error itself says, elasticsearch can't read the elastic-certificates.p12 file .

This is again already covered in the documentation that you have read, in the section https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html#deb-configuring

You need to make sure elasticsearch can read the file, do the following:

sudo chown root:elasticsearch /etc/elasticsearch/elastic-certificates.p12

Hey Thanks now elasticsearch is not getting crashed after doing this sudo chown root:elasticsearch /etc/elasticsearch/elastic-certificates.p12 and also i added chmod 777 /etc/elasticsearch/elastic-certificates.p12 as u mentioned . But this is what i am getting after

curl -X GET '10.128.*.**:9200/'

elastic-stage-vyakar:/etc/elasticsearch# curl -X GET '10.128.0.26:9200/'
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" 
charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" cha
rset=\"UTF-8\""}},"status":401}

i also tried this.

curl -u elastic:changeme http://10.128.0.26:9200/

{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""
}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

But status of elasticsearch is active.

this is log :

Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]: Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elastic-certificates.p12
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.nio.file.Files.newByteChannel(Files.java:373) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.nio.file.Files.newByteChannel(Files.java:424) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.nio.file.Files.newInputStream(Files.java:158) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:87) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:58) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:382) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.util.HashMap.computeIfAbsent(HashMap.java:1133) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:434) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:119) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:146) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:
?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:
45) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.3.0.jar:7.3
.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.3.0.jar:7.3
.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.3.0.jar:7.
3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.node.Node.<init>(Node.java:314) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.0.jar:7.3.0]
Aug 14 07:30:57 elastic-stage-vyakar elasticsearch[4748]:         ... 6 more
Aug 14 07:30:57 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Aug 14 07:30:57 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Please, please, please read the documentation. Almost all your questions in this post could be solved by reading the docs and following instructions instead of simply trying things and posting your errors.

You need to configure Elasticsearch. There is no user with changeme password so of course you cannot authenticate as such.

The doc we have been mentioning all this time ends with a link to

Learn how to configure Elasticsearch. There you can see https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-security.html. Follow the docs.

This is just 10 lines from the log from 2 hours ago, before you solved the problem you had with the file and as such are not useful now.

You need to start putting some effort in solving your problems on your side too, otherwise I'm sorry but I'll have to disengage as this is becoming very time consuming and is unfair for the rest of the users of this forum.

I am sorry that i am getting it very slow, i set password for elastic and kibana as well, but can you please tell me how to choose native realm i made a changes in elasticsearch.yml file here it is :

#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.audit.enabled: true
#xpack.monitoring.enabled: true
xpack.security.authc.realms.native: xpack:
  security:
    authc:
      realms:
        native:
          native1:
            order: 0
#---------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

can you please tell what is the right way to configure native realm.

kibana.yml:

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: kibanaindx

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"

If you don't use other realms ,the native realm is enabled by default as we say in our documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-native-realm.html

Just remove the

xpack.security.authc.realms.native:  xpack:
  security:
    authc:
      realms:
        native:
          native1:
            order: 0

part entirely. For reference, the correct way to define the native realm ,if need be , would be like :

xpack:
  security:
    authc:
      realms:
        native:
          native1:
            order: 0

as also shown in the same doc above.

I did every thing elasticsearch is up and running kibana is also up and running but after firing this curl -XEGT -u kibana '10.128.*.**:5601/' i got this.
Kibana server is not ready yet

kibana log:

     green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-16T06:52:34Z","tags":["status","plugin:index_lifecycle_management@7.3.0","info"],"pid":18478,"state":"green","message":"Status changed from
 yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-16T06:52:34Z","tags":["status","plugin:rollup@7.3.0","info"],"pid":18478,"state":"green","message":"Status changed from yellow to green - R
eady","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-16T06:52:34Z","tags":["status","plugin:remote_clusters@7.3.0","info"],"pid":18478,"state":"green","message":"Status changed from yellow to 
green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-16T06:52:34Z","tags":["status","plugin:cross_cluster_replication@7.3.0","info"],"pid":18478,"state":"green","message":"Status changed from 
yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-16T06:52:34Z","tags":["status","plugin:file_upload@7.3.0","info"],"pid":18478,"state":"green","message":"Status changed from yellow to gree
n - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-16T06:52:34Z","tags":["status","plugin:snapshot_restore@7.3.0","info"],"pid":18478,"state":"green","message":"Status changed from yellow to
 green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-16T06:52:34Z","tags":["info","monitoring","kibana-monitoring"],"pid":18478,"message":"Starting monitoring stats collection"}
{"type":"log","@timestamp":"2019-08-16T06:52:34Z","tags":["status","plugin:maps@7.3.0","info"],"pid":18478,"state":"green","message":"Status changed from yellow to green - Rea
dy","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-16T06:52:35Z","tags":["status","plugin:reporting@7.3.0","info"],"pid":18478,"state":"green","message":"Status changed from uninitialized to
 green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2019-08-16T06:52:35Z","tags":["status","plugin:spaces@7.3.0","error"],"pid":18478,"state":"red","message":"Status changed from yellow to red - acti
on [indices:admin/get] is unauthorized for user [kibana]: [security_exception] action [indices:admin/get] is unauthorized for user [kibana]","prevState":"yellow","prevMsg":"Wa
iting for Elasticsearch"}
{"type":"log","@timestamp":"2019-08-16T06:52:36Z","tags":["fatal","root"],"pid":18478,"message":"{ [security_exception] action [indices:admin/get] is unauthorized for user [ki
bana] :: {\"path\":\"/kibanaindx\",\"query\":{},\"statusCode\":403,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\
\\"action [indices:admin/get] is unauthorized for user [kibana]\\\"}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"action [indices:admin/get] is unauthorized for
 user [kibana]\\\"},\\\"status\\\":403}\"}\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:315:15)\n    at checkRespForFailure (/usr/share/k
ibana/node_modules/elasticsearch/src/lib/transport.js:274:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\
n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at en
dReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 403,\n  displayName: 'AuthorizationException',\n  mess
age:\n   'action [indices:admin/get] is unauthorized for user [kibana]: [security_exception] action [indices:admin/get] is unauthorized for user [kibana]',\n  path: '/kibanain
dx',\n  query: {},\n  body:\n   { error:\n      { root_cause: [Array],\n        type: 'security_exception',\n        reason:\n         'action [indices:admin/get] is unauthori
zed for user [kibana]' },\n     status: 403 },\n  statusCode: 403,\n  response:\n   '{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:
admin/get] is unauthorized for user [kibana]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:admin/get] is unauthorized for user [kibana]\"},\"status\":403}',
\n  toString: [Function],\n  toJSON: [Function],\n  isBoom: true,\n  isServer: false,\n  data: null,\n  output:\n   { statusCode: 403,\n     payload:\n      { message:\n      
   'action [indices:admin/get] is unauthorized for user [kibana]: [security_exception] action [indices:admin/get] is unauthorized for user [kibana]',\n        statusCode: 403,
\n        error: 'Forbidden' },\n     headers: {} },\n  reformat: [Function],\n  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/forbidden' }"}
{"type":"log","@timestamp":"2019-08-16T06:52:54Z","tags":["info","plugins-system"],"pid":18591,"message":"Setting up [1] plugins: [translations]"}
{"type":"log","@timestamp":"2019-08-16T06:52:54Z","tags":["info","plugins","translations"],"pid":18591,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2019-08-16T06:52:54Z","tags":["info","plugins-system"],"pid":18591,"message":"Starting [1] plugins: [translations]"}

kibana.yml :

server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: 10.128.0.26
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name.  This is used for display purposes.
server.name: testing
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: http://10.128.0.26:9200
# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: kibanaindx
# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full
# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500
# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000
# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]
# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000
# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000
# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false
# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid
# Enables you specify a file where Kibana stores log output.
logging.dest: /etc/kibana/kibana.log
xpack.security.encryptionKey: "something_at_least_32_characters"
xpack.reporting.encryptionKey: "kibana"
#xpack.monitoring.enabled: true
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false
# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false
# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false
# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000
# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN . 
#i18n.locale: "en"

There is no point usually doing a curl to Kibana. You most probably want to use Kibana as a user interface so you'd normally access that with your browser.

Also, kibana is the user that Kibana uses to communicate to Elasticsearch, and should not be the user you use to authenticate to Kibana as an end user.

So, open up your browser, type http://<YOUR IP ADDRESS>:5601 , and when prompted for a username and password, use elastic as the username and the password you have set for the elastic user.

yes i did that too and its printing same message Kibana server is not ready yet in the web browser. Also i am getting this message before username and password. What else i can do.

should i do this .

Set up roles and users to control access to Elasticsearch.

For example, to grant John Doe full access to all indices that match the pattern events* and enable him to create visualizations and dashboards for those indices in Kibana, you could create an events_admin role and assign the role to a new johndoe user.

curl -XPOST -u elastic 'localhost:9200/_security/role/events_admin' -H "Content-Type: application/json" -d '{
  "indices" : [
    {
      "names" : [ "events*" ],
      "privileges" : [ "all" ]
    },
    {
      "names" : [ ".kibana*" ],
      "privileges" : [ "manage", "read", "index" ]
    }
  ]
}'

curl -XPOST -u elastic 'localhost:9200/_security/user/johndoe' -H "Content-Type: application/json" -d '{
  "password" : "userpassword",
  "full_name" : "John Doe",
  "email" : "john.doe@anony.mous",
  "roles" : [ "events_admin" ]
}'

Yes, at some point, but not for this exact problem you are seeing now.

Try using a different browser or try restarting your browser, or try clearing your cache and cookies. When you get prompted for username/password again, use the elastic user and its password.

Really thanx, now everything is working fine. :grinning::grinning::grinning:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.