Elasticsearch doesn’t start: status=1/FAILURE (TLS/x-pack)

Elastic Stack Elasticsearch
1

Hello!

I've tried to follow this list:

Install a single ES node.
Install X-Pack on that node.
Run setup-passwords on that node.
Add additional nodes (ES + X-Pack).
Enable TLS for node transport.
Enable TLS for HTTP.

I'm using this manual https://www.elastic.co/guide/en/elasticsearch/reference/current/installing-xpack-es.html
I've setuped 1node in test purposes , than installed and setuped x-pack. After I've installed additional 2 node. And if I uncomment strings in first node, for example:

#discovery.zen.ping.unicast.hosts: ["192.168.222.232, 192.168.222.233"]

or

         #xpack.security.transport.ssl.enabled: true
        #xpack.security.transport.ssl.verification_mode: certificate
        #xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/master.p12
        #xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/master.p12

my elastic goes down ( :

Mar 11 14:46:33 master systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Mar 11 14:46:33 master systemd[1]: elasticsearch.service: Unit entered failed state.
Mar 11 14:46:33 master systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

[2018-03-11T15:19:20,930][ERROR][o.e.b.Bootstrap          ] [master] node validation exception

And I don't understand, what I should do.....(

2

Can you paste in a larger part of your log so that we can see the error?

3

Yes, I did .
I'll paste log later

4

Right below:
[2018-03-11T15:19:20,930][ERROR][o.e.b.Bootstrap ] [master] node validation exception

you should see that bootstrap checks that have failed. This will give us a proper indication of what goes wrong

1 Like
5 
[2018-03-12T22:23:17,470][INFO ][o.e.n.Node               ] [master] initializing ...
[2018-03-12T22:23:18,270][INFO ][o.e.e.NodeEnvironment    ] [master] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [8.6gb], net total_space [11.9gb], types [btrfs]
[2018-03-12T22:23:18,270][INFO ][o.e.e.NodeEnvironment    ] [master] heap size [247.5mb], compressed ordinary object pointers [true]
[2018-03-12T22:23:18,412][INFO ][o.e.n.Node               ] [master] node name [master], node ID [KC0RJjSzQhiagzzhHcDjag]
[2018-03-12T22:23:18,412][INFO ][o.e.n.Node               ] [master] version[6.2.2], pid[384], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.0-6-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-03-12T22:23:18,412][INFO ][o.e.n.Node               ] [master] JVM arguments [-Xms256m, -Xmx256m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.2ED2eX9r, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [aggs-matrix-stats]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [analysis-common]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [ingest-common]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-expression]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-mustache]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-painless]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [mapper-extras]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [parent-join]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [percolator]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [rank-eval]
[2018-03-12T22:23:24,190][INFO ][o.e.p.PluginsService     ] [master] loaded module [reindex]
[2018-03-12T22:23:24,191][INFO ][o.e.p.PluginsService     ] [master] loaded module [repository-url]
[2018-03-12T22:23:24,191][INFO ][o.e.p.PluginsService     ] [master] loaded module [transport-netty4]
[2018-03-12T22:23:24,191][INFO ][o.e.p.PluginsService     ] [master] loaded module [tribe]
[2018-03-12T22:23:24,191][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [x-pack-core]
[2018-03-12T22:23:24,191][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [x-pack-deprecation]
[2018-03-12T22:23:24,191][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [x-pack-graph]
[2018-03-12T22:23:24,191][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [x-pack-logstash]
[2018-03-12T22:23:24,192][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [x-pack-ml]
[2018-03-12T22:23:24,192][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [x-pack-monitoring]
[2018-03-12T22:23:24,192][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [x-pack-security]
[2018-03-12T22:23:24,192][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [x-pack-upgrade]
[2018-03-12T22:23:24,192][INFO ][o.e.p.PluginsService     ] [master] loaded plugin [x-pack-watcher]
[2018-03-12T22:23:27,885][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/816] [Main.cc@128] controller (64 bit): Version 6.2.2 (Build 61c43548e5e0f2) Copyright (c) 2018 Elasticsearch BV
[2018-03-12T22:23:29,882][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-03-12T22:23:30,434][INFO ][o.e.d.DiscoveryModule    ] [master] using discovery type [zen]
[2018-03-12T22:23:31,352][INFO ][o.e.n.Node               ] [master] initialized
[2018-03-12T22:23:31,352][INFO ][o.e.n.Node               ] [master] starting ...
[2018-03-12T22:23:31,590][INFO ][o.e.t.TransportService   ] [master] publish_address {192.168.222.232:9300}, bound_addresses {192.168.222.232:9300}
[2018-03-12T22:23:31,629][INFO ][o.e.b.BootstrapChecks    ] [master] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-03-12T22:23:34,724][INFO ][o.e.c.s.MasterService    ] [master] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {master}{KC0RJjSzQhiagzzhHcDjag}{6njvab3JSzuhSdFSM6Cg4Q}{192.168.222.232}{192.168.222.232:9300}{ml.machine_memory=2101534720, ml.max_open_jobs=20, ml.enabled=true}
[2018-03-12T22:23:34,729][INFO ][o.e.c.s.ClusterApplierService] [master] new_master {master}{KC0RJjSzQhiagzzhHcDjag}{6njvab3JSzuhSdFSM6Cg4Q}{192.168.222.232}{192.168.222.232:9300}{ml.machine_memory=2101534720, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {master}{KC0RJjSzQhiagzzhHcDjag}{6njvab3JSzuhSdFSM6Cg4Q}{192.168.222.232}{192.168.222.232:9300}{ml.machine_memory=2101534720, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-03-12T22:23:34,797][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [master] publish_address {192.168.222.232:9200}, bound_addresses {[::]:9200}
[2018-03-12T22:23:34,798][INFO ][o.e.n.Node               ] [master] started
[2018-03-12T22:23:37,601][INFO ][o.e.l.LicenseService     ] [master] license [5068e9b2-4634-413c-8dfe-3bfe8d47409f] mode [trial] - valid
[2018-03-12T22:23:37,608][WARN ][o.e.l.LicenseService     ] [master]
#
# License [will expire] on [Tuesday, April 03, 2018]. If you have a new license, please update it.
# Otherwise, please reach out to your support contact.
#
6 
# Commercial plugins operate with reduced functionality on license expiration:
# - security
#  - Cluster health, cluster stats and indices stats operations are blocked
#  - All data operations (read and write) continue to work
# - watcher
#  - PUT / GET watch APIs are disabled, DELETE watch API continues to work
#  - Watches execute and write to the history
#  - The actions of the watches don't execute
# - monitoring
#  - The agent will stop collecting cluster and indices metrics
#  - The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]
# - graph
#  - Graph explore APIs are disabled
# - ml
#  - Machine learning APIs are disabled
# - logstash
#  - Logstash will continue to poll centrally-managed pipelines
# - deprecation
#  - Deprecation APIs are disabled
# - upgrade
#  - Upgrade API is disabled
[2018-03-12T22:23:37,619][INFO ][o.e.g.GatewayService     ] [master] recovered [14] indices into cluster_state
[2018-03-12T22:23:41,490][INFO ][o.e.x.s.a.s.m.NativeRoleMappingStore] [master] The security index is not yet available - no role mappings can be loaded
[2018-03-12T22:23:42,497][INFO ][o.e.c.m.MetaDataCreateIndexService] [master] [.monitoring-es-6-2018.03.12] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [doc]
[2018-03-12T22:23:54,074][INFO ][o.e.c.r.a.AllocationService] [master] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.monitoring-es-6-2018.03.09][0]] ...]).
[2018-03-12T22:24:05,453][INFO ][o.e.c.m.MetaDataCreateIndexService] [master] [.monitoring-kibana-6-2018.03.12] creating index, cause [auto(bulk api)], templates [.monitoring-kibana], shards [1]/[0], mappings [doc]
[2018-03-12T22:24:05,601][INFO ][o.e.c.r.a.AllocationService] [master] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-kibana-6-2018.03.12][0]] ...]).
[2018-03-12T22:24:49,189][INFO ][o.e.c.m.MetaDataCreateIndexService] [master] [.watcher-history-7-2018.03.12] creating index, cause [auto(bulk api)], templates [.watch-history-7], shards [1]/[0], mappings [doc]
[2018-03-12T22:24:49,401][INFO ][o.e.c.r.a.AllocationService] [master] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.watcher-history-7-2018.03.12][0]] ...]).
[2018-03-12T22:24:49,461][INFO ][o.e.c.m.MetaDataMappingService] [master] [.watcher-history-7-2018.03.12/61AZoqRBSGK_saEDEfyyLA] update_mapping [doc]
[2018-03-12T22:24:49,475][INFO ][o.e.c.m.MetaDataMappingService] [master] [.watcher-history-7-2018.03.12/61AZoqRBSGK_saEDEfyyLA] update_mapping [doc]
[2018-03-12T22:24:49,565][INFO ][o.e.c.m.MetaDataMappingService] [master] [.watcher-history-7-2018.03.12/61AZoqRBSGK_saEDEfyyLA] update_mapping [doc]
[2018-03-12T22:57:10,428][INFO ][o.e.n.Node               ] [master] stopping ...
[2018-03-12T22:57:10,467][INFO ][o.e.x.w.WatcherService   ] [master] stopping watch service, reason [shutdown initiated]
[2018-03-12T22:57:10,467][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/816] [Main.cc@168] Ml controller exiting
[2018-03-12T22:57:10,468][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
[2018-03-12T22:57:11,381][INFO ][o.e.n.Node               ] [master] stopped
[2018-03-12T22:57:11,381][INFO ][o.e.n.Node               ] [master] closing ...
[2018-03-12T22:57:11,406][INFO ][o.e.n.Node               ] [master] closed
[2018-03-12T22:57:13,085][INFO ][o.e.n.Node               ] [master] initializing ...
[2018-03-12T22:57:13,194][INFO ][o.e.e.NodeEnvironment    ] [master] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [8.7gb], net total_space [11.9gb], types [btrfs]
[2018-03-12T22:57:13,194][INFO ][o.e.e.NodeEnvironment    ] [master] heap size [247.5mb], compressed ordinary object pointers [true]
[2018-03-12T22:57:13,248][INFO ][o.e.n.Node               ] [master] node name [master], node ID [KC0RJjSzQhiagzzhHcDjag]
[2018-03-12T22:57:13,249][INFO ][o.e.n.Node               ] [master] version[6.2.2], pid[7443], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.0-6-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-03-12T22:57:13,249][INFO ][o.e.n.Node               ] [master] JVM arguments [-Xms256m, -Xmx256m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.c1KRiW3p, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-03-12T22:57:14,704][ERROR][o.e.b.Bootstrap          ] Exception
7 
[2018-03-12T22:57:14,704][ERROR][o.e.b.Bootstrap          ] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:563) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:505) ~[elasticsearch-6.2.2.jar:6.2.2]
       
        at org.elasticsearch.node.Node.<init>(Node.java:303) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.2.jar:6.2.2]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        

Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:72) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:412) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_151]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:464) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:91) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:127) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
 

Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/master.p12
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
        at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_151]
  


[2018-03-12T22:57:14,721][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [master] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.2.jar:6.2.2]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.2.jar:6.2.2]
 
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.core.XPackPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:563) ~[elasticsearch-6.2.2.jar:6.2.2]
    
 
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_151]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:554) ~[elasticsearch-6.2.2.jar:6.2.2]
        
        at org.elasticsearch.node.Node.<init>(Node.java:303) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.2.jar:6.2.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
      
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
        at org.elasticsearch.xpack.core.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:72) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:412) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[?:1.8.0_151]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:464) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:91) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.<init>(XPackPlugin.java:127) ~[?:?]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        

Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/master.p12
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
8 
root@master:/var/log/elasticsearch# systemctl status  elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2018-03-12 22:57:14 +03; 1min 2s ago
     Docs: http://www.elastic.co
  Process: 7443 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1
 Main PID: 7443 (code=exited, status=1/FAILURE)

Mar 12 22:57:11 master systemd[1]: Started Elasticsearch.
Mar 12 22:57:14 master systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Mar 12 22:57:14 master systemd[1]: elasticsearch.service: Unit entered failed state.
Mar 12 22:57:14 master systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
9

This is the key part of the log.
What are the permissions on that file?

1 Like
10

-rw------- 1 root elasticsearch

I've changed permission to rx and java.nio.file.AccessDeniedException - disappeared, but it didn't help - elastic doesn't start

11 
/var/log/elasticsearch# cat my-application.log
[2018-03-13T21:43:26,626][INFO ][o.e.n.Node               ] [master] initializing ...
[2018-03-13T21:43:26,734][INFO ][o.e.e.NodeEnvironment    ] [master] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [8.7gb], net total_space [11.9gb], types [btrfs]
[2018-03-13T21:43:26,735][INFO ][o.e.e.NodeEnvironment    ] [master] heap size [247.5mb], compressed ordinary object pointers [true]
[2018-03-13T21:43:26,803][INFO ][o.e.n.Node               ] [master] node name [master], node ID [KC0RJjSzQhiagzzhHcDjag]
[2018-03-13T21:43:26,804][INFO ][o.e.n.Node               ] [master] version[6.2.2], pid[2882], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.0-6-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-03-13T21:43:26,804][INFO ][o.e.n.Node               ] [master] JVM arguments [-Xms256m, -Xmx256m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.FaQnYx8c, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-03-13T21:43:29,633][INFO ][o.e.p.PluginsService     ] [master] loaded module [aggs-matrix-stats]
[2018-03-13T21:43:29,633][INFO ][o.e.p.PluginsService     ] [master] loaded module [analysis-common]
[2018-03-13T21:43:29,633][INFO ][o.e.p.PluginsService     ] [master] loaded module [ingest-common]
[2018-03-13T21:43:29,633][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-expression]
[2018-03-13T21:43:29,633][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-mustache]
[2018-03-13T21:43:29,633][INFO ][o.e.p.PluginsService     ] [master] loaded module [lang-painless]

[2018-03-13T21:43:31,981][ERROR][o.e.b.Bootstrap          ] Exception
java.lang.IllegalArgumentException: unknown setting [transport.port] did you mean any of [transport.tcp.port, transport.type]?


[2018-03-13T21:43:31,987][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [master] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [transport.port] did you mean any of [transport.tcp.port, transport.type]?

Caused by: java.lang.IllegalArgumentException: unknown setting [transport.port] did you mean any of [transport.tcp.port, transport.type]?
   
root@master:/var/log/elasticsearch#
12 
/etc/elasticsearch# cat elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: master
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.222.232
#
# Set a custom port for HTTP:
#
http.port: 9200
http.host: 0.0.0.0
#####http.host: 192.168.222.232
transport.host: 192.168.222.232
transport.port: 9300
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["192.168.222.232, 192.168.222.233:9300"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.audit.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/master.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/master.p12
13

Well, it clearly did help, because it resolved that error, and now you have a different error.

This message spells out exactly what is wrong. transport.port is not a valid setting. The correct setting is transport.tcp.port.

1 Like
14

)))))

15 
   /var/log/elasticsearch# curl -s 'localhost:9200/_cat/nodes?pretty'  -u elastic:pass
192.168.222.232 54 89 13 0.14 0.30 0.27 mdi * master

 [2018-03-15T00:24:12,930][INFO ][o.e.d.DiscoveryModule    ] [master] using discovery type [zen]
    [2018-03-15T00:24:13,692][INFO ][o.e.n.Node               ] [master] initialized
    [2018-03-15T00:24:13,692][INFO ][o.e.n.Node               ] [master] starting ...
    [2018-03-15T00:24:13,852][INFO ][o.e.t.TransportService   ] [master] publish_address {192.168.222.232:9300}, bound_addresses {192.168.222.232:9300}
    [2018-03-15T00:24:13,900][INFO ][o.e.b.BootstrapChecks    ] [master] bound or publishing to a non-loopback address, enforcing bootstrap checks
    [2018-03-15T00:24:14,040][WARN ][o.e.d.z.UnicastZenPing   ] [master] failed to resolve host [192.168.222.232, 192.168.222.233:9300]
    java.net.UnknownHostException: 192.168.222.232, 192.168.222.233: Name or service not known

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
[2018-03-15T00:24:17,092][INFO ][o.e.c.s.MasterService    ] [master] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {master}{KC0RJjSzQhiagzzhHcDjag}{8mMWffjgSUGmhqTBxba3Aw}{192.168.222.232}{192.168.222.232:9300}{ml.machine_memory=2101534720, ml.max_open_jobs=20, ml.enabled=true}
[2018-03-15T00:24:17,098][INFO ][o.e.c.s.ClusterApplierService] [master] new_master {master}{KC0RJjSzQhiagzzhHcDjag}{8mMWffjgSUGmhqTBxba3Aw}{192.168.222.232}{192.168.222.232:9300}{ml.machine_memory=2101534720, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {master}{KC0RJjSzQhiagzzhHcDjag}{8mMWffjgSUGmhqTBxba3Aw}{192.168.222.232}{192.168.222.232:9300}{ml.machine_memory=2101534720, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-03-15T00:24:17,143][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [master] publish_address {192.168.222.232:9200}, bound_addresses {[::]:9200}
[2018-03-15T00:24:17,143][INFO ][o.e.n.Node               ] [master] started
[2018-03-15T00:24:17,973][INFO ][o.e.x.s.a.AuthenticationService] [master] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-03-15T00:24:17,989][INFO ][o.e.x.s.a.AuthenticationService] [master] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-03-15T00:24:18,169][INFO ][o.e.x.s.a.AuthenticationService] [master] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-03-15T00:24:18,810][INFO ][o.e.x.s.a.AuthenticationService] [master] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-03-15T00:24:18,980][INFO ][o.e.x.s.a.AuthenticationService] [master] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-03-15T00:24:19,114][INFO ][o.e.x.s.a.AuthenticationService] [master] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2018-03-15T00:24:19,142][INFO ][o.e.l.LicenseService     ] [master] license [5068e9b2-4634-413c-8dfe-3bfe8d47409f] mode [trial] - valid
[2018-03-15T00:24:19,148][WARN ][o.e.l.LicenseService     ] [master]
#
# License [will expire] on [Tuesday, April 03, 2018]. If you have a new license, please update it.
# Otherwise, please reach out to your support contact.
#


# - upgrade
#  - Upgrade API is disabled
[2018-03-15T00:24:19,158][INFO ][o.e.g.GatewayService     ] [master] recovered [25] indices into cluster_state
[2018-03-15T00:24:22,505][INFO ][o.e.c.r.a.AllocationService] [master] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-es-6-2018.03.09][0], [.monitoring-kibana-6-2018.03.09][0], [.monitoring-alerts-6][0], [.watcher-history-7-2018.03.09][0]] ...]).
root@master:/var/log/elasticsearch#
16

I've seeing error and corrected it

[quote="manst, post:12, topic:123498"]
discovery.zen.ping.unicast.hosts: ["192.168.222.232:9300", "192.168.222.233:9300"]

Now my first node see the second.

And next bulb of errors:

[INFO ][o.e.d.z.ZenDiscovery     ] [master] failed to send join request to master [{data}{KC0RJjSzQhiagzzhHcDjag}{kC1Lw4_2RV6H8ebfXMgXGw}{192.168.222.233}{192.168.222.233:9300}{ml.machine_memory=1722236928, ml.max_open_jobs=20, ml.enabled=true}], reason [RemoteTransportException[[data][192.168.222.233:9300][internal:discovery/zen/join]]; nested: IllegalArgumentException[can't add node {master}{KC0RJjSzQhiagzzhHcDjag}{0lx-KxDlS5iVU-TI1xhMew}{192.168.222.232}{192.168.222.232:9300}{ml.machine_memory=2101534720, ml.max_open_jobs=20, ml.enabled=true}, found existing node {data}{KC0RJjSzQhiagzzhHcDjag}{kC1Lw4_2RV6H8ebfXMgXGw}{192.168.222.233}{192.168.222.233:9300}{ml.machine_memory=1722236928, ml.max_open_jobs=20, ml.enabled=true} with the same id but is a different node instance]; ]

solution for this error (you must deleted contents of data folder(/var/lib/elasticsearch/nodes/0) and restarted both the servers ):

Than I should change passwords , because i've had error:

/var/log/elasticsearch# curl -s 'localhost:9200/_cat/nodes?pretty'  -u elastic:pass
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "failed to authenticate user [elastic]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "failed to authenticate user [elastic]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
    }
  },
  "status" : 401
17

Urra))))):

/var/log/elasticsearch# curl -s 'localhost:9200/_cat/nodes?pretty'  -u elastic:pass
192.168.222.233 74 45 22 0.24 0.33 0.26 mdi * data
192.168.222.232 55 62  5 0.18 0.14 0.13 mdi - master

Thanks everyone a lot for your help.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.