Elasticsearch active is running but port 9200 doesn't listen

Nguyen_Hieu · April 27, 2019, 9:46am

Hi everyone,

I start elasticsearch and check status, it's running. But when I use curl to access, I receive an message: Connection refused.
Here is my code when I check in command line:

                 root@hieu: service elasticsearch start
                 root@hieu: systemctl -l status elasticsearch
            ● elasticsearch.service - Elasticsearch
               Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendo
               Active: active (running) since Sat 2019-04-27 16:34:31 +07; 1s ago
                 Docs: http://www.elastic.co
             Main PID: 4980 (java)
                Tasks: 15 (limit: 4535)
               CGroup: /system.slice/elasticsearch.service
                       └─4980 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSIni

            Thg 4 27 16:34:31 hieu-Latitude-E6420 systemd[1]: Started Elasticsearch.
            Thg 4 27 16:34:31 hieu-Latitude-E6420 elasticsearch[4980]: warning: Falling back
            Thg 4 27 16:34:32 hieu-Latitude-E6420 elasticsearch[4980]: OpenJDK 64-Bit Server
            Thg 4 27 16:34:32 hieu-Latitude-E6420 elasticsearch[4980]: OpenJDK 64-Bit Server

            root@hieu-Latitude-E6420:/home/hieu# curl -XGET 127.0.0.1:9200
            curl: (7) Failed to connect to 127.0.0.1 port 9200: Connection refused

Check netstat:

root@hieu-Latitude-E6420:/home/hieu# netstat -tulnp | grep 9200

and it's not respond anything.

In my elasticsearch.yml:

    network.host: localhost
    http.port: 9200

Thanks.

dadoonet · April 27, 2019, 11:14am

What are elasticsearch logs?

Nguyen_Hieu · April 27, 2019, 11:52am

Thanks for asked, this is what I saw in elasticsearch.log:

[2019-04-27T18:45:52,608][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [HzcmFIK] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/global-0.st]]; nested: IOException[failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/global-0.st]]; nested: IllegalArgumentException[Unexpected field [cluster_uuid_committed]];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163)

dadoonet · April 27, 2019, 1:12pm

That's probably why. Could you share the full log please ?

And explain what you exactly did. Like did you install a new version...

ywelsch · April 27, 2019, 4:27pm

based on the exception here, it looks like you upgraded to elasticsearch version 7, and then downgraded to version 6. Downgrades are not possible.

Nguyen_Hieu · April 28, 2019, 2:49am

In elasticsearch log ( full log):

[2019-04-28T09:36:14,460][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [HzcmFIK] [controller/21576] [Main.cc@109] controller (64 bit): Version 6.7.1 (Build e1f492de67a719) Copyright (c) 2019 Elasticsearch BV
[2019-04-28T09:36:15,860][DEBUG][o.e.a.ActionModule       ] [HzcmFIK] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-04-28T09:36:15,911][ERROR][o.e.g.GatewayMetaState   ] [HzcmFIK] failed to read local state, exiting...
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/global-0.st]
	at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.node.Node.<init>(Node.java:499) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.node.Node.<init>(Node.java:266) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.7.1.jar:6.7.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.7.1.jar:6.7.1]
Caused by: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/global-0.st]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.7.1.jar:6.7.1]
	... 15 more
Caused by: java.lang.IllegalArgumentException: Unexpected field [cluster_uuid_committed]
	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1270) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.7.1.jar:6.7.1]
	... 15 more
[2019-04-28T09:36:15,938][ERROR][o.e.b.Bootstrap          ] [HzcmFIK] Exception
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/global-0.st]
	at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.7.1.jar:6.7.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.7.1.jar:6.7.1]
Caused by: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/global-0.st]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.7.1.jar:6.7.1]
	... 15 more
Caused by: java.lang.IllegalArgumentException: Unexpected field [cluster_uuid_committed]
	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1270) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.7.1.jar:6.7.1]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.7.1.jar:6.7.1]
	... 15 more

[2019-04-28T09:36:15,960][INFO ][o.e.x.m.p.NativeController] [HzcmFIK] Native controller process has stopped - no new native processes can be started

Nguyen_Hieu · April 28, 2019, 2:50am

I use this links to install
(https://linuxize.com/post/how-to-install-elasticsearch-on-ubuntu-18-04/)

install java8 -> apt update -> Import the repository’s GPG -> add the Elasticsearch repository to the system -> update -> install elasticsearch

I also set path for JAVA_HOME:

root@hieu-Latitude-E6420:/etc/apt/sources.list.d# echo $JAVA_HOME
/usr/lib/jvm/open-jdk

But still got this warning:

warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME

Do you think I should remove and install new version?
Currently, I'm use elasticsearch version 6.7.1

Nguyen_Hieu · April 28, 2019, 3:06am

I used: apt-get remove --purge elasticsearch to remove version before and install ver 6.7.1 . And I following this tutorial to install:
https://linuxize.com/post/how-to-install-elasticsearch-on-ubuntu-18-04/

Nguyen_Hieu · April 28, 2019, 8:32am

Thank two guys, I just change $JAVA_HOME from/usr/lib/jvm/java to /usr/lib/jvm/java-8-openjdk-amd64 (I check list of file in jvm folder and saw java-8-openjdk-amd64 instead of java ), then restart. I don't know what's wrong but it's work now.

Thank for your help again!

system · May 26, 2019, 8:32am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.