Elasticsearch could not find Java and now won't start

sc1 · March 26, 2019, 5:03pm

Hello,

I recently updated 4 Elasticsearch nodes and I now have an error on 2 nodes, I have looked online about how to resolve it or specify but I can't find anything on the specific problem I am having (perhaps I am searching for the wrong thing).

I am not a Java expert but here is the issue:

root@vComputerYacho:/usr/share/elasticsearch/bin# sudo service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-03-26 16:20:38 GMT; 37min ago
     Docs: http://www.elastic.co
  Process: 27459 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 27459 (code=exited, status=1/FAILURE)

Mar 26 16:20:14 vComputerYacho systemd[1]: Started Elasticsearch.
Mar 26 16:20:14 vComputerYacho elasticsearch[27459]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Mar 26 16:20:38 vComputerYacho systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 16:20:38 vComputerYacho systemd[1]: elasticsearch.service: Unit entered failed state.
Mar 26 16:20:38 vComputerYacho systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Of the 4 nodes I updated, 2/3 in one cluster are working fine. 1 which isn't clustered is having this issue. Could someone be as kind to advise on how I can resolve this or explain how an update caused this?

uhlhosting · March 26, 2019, 5:59pm

I got the same issue:

**●** elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: **failed** (Result: exit-code) since Tue 2019-03-26 18:57:24 CET; 615ms ago
Docs: http://www.elastic.co
Process: 23351 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet **(code=exited, status=1/FAI**
Main PID: 23351 (code=exited, status=1/FAILURE)
Mar 26 18:57:14 logs systemd[1]: Started Elasticsearch.
Mar 26 18:57:14 logs elasticsearch[23351]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Mar 26 18:57:24 logs systemd[1]: **elasticsearch.service: Main process exited, code=exited, status=1/FAILURE**
Mar 26 18:57:24 logs systemd[1]: **elasticsearch.service: Failed with result 'exit-code'.**

While upon

root@logs:/etc/elasticsearch# echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle/jre

duckulaj · March 26, 2019, 10:38pm

Exactly the same problem here

Blockquote
Mar 26 22:29:05 HP-LINUX-MINT elasticsearch[14306]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME

Blockquote
echo $JAVA_HOME
/opt/java/jdk1.8.0_191

Version 6.7.0 and ingest-plugin 6.7.0

muhamedHassan · March 27, 2019, 7:48am

same problem here
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME

boxclever38 · March 27, 2019, 4:35pm

This happened to me too, but the JAVA HOME error was a red herring. Check your elasticsearch logs and see if there is a plugin conflict. Mine was the geoip-ingest module after upgrading to 6.7
Removing the module allowed elasticsearch to start normally.

sc1 · March 27, 2019, 5:18pm

Thanks Chris, I did have think it was that and went ahead and uninstalled. But I can see from the latest 6.7 documentation, the plugin is now standard so I'm not even sure how to proceed.

I have troubleshooted though and here is the logs:

[2019-03-27T16:59:32,887][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [vComputerYacho] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.7.0.jar:6.7.0]
Caused by: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:78) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.0.jar:6.7.0]
	... 6 more
Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:102) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.0.jar:6.7.0]

So yes, it looks like the geoip-ingest module is at fault. I have tried to uninstall it:

root@vComputerYacho:/# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove  ingest-geoip
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
-> removing [ingest-geoip]...
ERROR: plugin [ingest-geoip] not found; run 'elasticsearch-plugin list' to get list of installed plugins

When I try list modules to see if it still exists it doesnt return anything so I can't actually tell if it's installed or not.

root@vComputerYacho:/# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin list
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME

boxclever38 · March 27, 2019, 6:00pm

I think you're having the same experience as me after uninstalling ingest-geoip. The line that was my clue is:

Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]

The folder /etc/elasticsearch/ingest-geoip still exists and elasticsearch doesn't like it. I just renamed the folder rather than deleting it (I'm still a little too green at this to feel comfortable just deleting). After renaming the folder to ingest-geoip-backup elasticsearch started A-OK.

Let me know if it works for you.

boxclever38 · March 27, 2019, 6:01pm

I'd also be curios to know about the nodes that worked OK. How did they compare?

mukai · March 27, 2019, 10:28pm

You may need "remove --purge".

Elasticsearch 6.7.0 checks that there are no leftover geoip database files in the plugin configuration directory because the new module does not use them. Therefore, remove the ingest-geoip plugin prior to upgrading to 6.7.0 with the --purge option to also delete the old database files:
> ./bin/elasticsearch-plugin remove --purge ingest-geoip
Otherwise you will see the following error message upon startup (assuming /etc/elasticsearch/ingest-geoip as the plugin configuration directory):
expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.7.html#_literal_ingest_geoip_literal_and_literal_ingest_user_agent_literal_are_no_longer_distributed_as_plugins

In my case, I could solve it by executing the command.

warkolm · March 28, 2019, 3:54am

As @mukai mentions, you do need to run a purge on the ingest-geoip plugin.

See https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.7.html#_literal_ingest_geoip_literal_and_literal_ingest_user_agent_literal_are_no_longer_distributed_as_plugins

sc1 · March 28, 2019, 11:48am

Thanks for the replies @warkolm and @mukai

The purge option didn't actually fix it for me as the documentation suggested. When I ran the purge function, because I had previously performed the remove function, this was the result:

root@vComputerYacho:~# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove --purge  geoip-ingest
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
-> removing [geoip-ingest]...
ERROR: plugin [geoip-ingest] not found; run 'elasticsearch-plugin list' to get list of installed plugins

I checked the logs and could see the issue was in /etc/elasticsearch/ingest-geoip/

rm -rf /etc/elasticsearch/ingest-geoip/

After performing that, my 1st node was working again. I have also replicated on the other node, which I hadn't troubleshoot. The purge option alone is not enough, you still have to rm -rf the folder in the/etc/elasticsearch

Thanks again for the help!

uhlhosting · March 29, 2019, 9:25pm

sasd

uhlhosting · March 29, 2019, 9:28pm

The option to remove ingest plugin is not working since the plugin list shows none installed.
Yet I get this:

The command to install a plugin then throws this:
ERROR: Unknown plugin geoip-ingest

Anyone any ideas?

I wonder how people with OSS managed to make $ from so much ignorance.

root@logs:/etc/elasticsearch# locate ingest-geoip
/etc/elasticsearch/ingest-geoip
/usr/share/elasticsearch/modules/ingest-geoip
/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-ASN.mmdb
/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-City.mmdb
/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-Country.mmdb
/usr/share/elasticsearch/modules/ingest-geoip/geoip2-2.9.0.jar
/usr/share/elasticsearch/modules/ingest-geoip/ingest-geoip-6.7.0.jar
/usr/share/elasticsearch/modules/ingest-geoip/jackson-annotations-2.8.11.jar
/usr/share/elasticsearch/modules/ingest-geoip/jackson-databind-2.8.11.3.jar
/usr/share/elasticsearch/modules/ingest-geoip/maxmind-db-1.2.2.jar
/usr/share/elasticsearch/modules/ingest-geoip/plugin-descriptor.properties
/usr/share/elasticsearch/modules/ingest-geoip/plugin-security.policy
root@logs:/etc/elasticsearch# rm -rf /etc/elasticsearch/ingest-geoip
root@logs:/etc/elasticsearch# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove --purge geoip-ingest
-&gt; removing [geoip-ingest]...
ERROR: plugin [geoip-ingest] not found; run 'elasticsearch-plugin list' to get list of installed plugins
root@logs:/etc/elasticsearch#

And from the logs:
https://pastebin.com/DYdj90iE

After a while decided to manually remove all plugins and modules, and reinstalled elasticsearch, this helped, while meanwhile, logstash fails with:

[2019-03-29T23:02:47,796][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:03:07,668][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:03:28,046][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:03:47,884][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:04:08,362][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:04:28,204][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:04:48,694][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:05:08,807][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

jasontedor · March 30, 2019, 12:46pm

Sorry that you're having trouble.

As the docs say:

Therefore, remove the ingest-geoip plugin prior to upgrading to 6.7.0 with the --purge option to also delete the old database files

Note that this should be done prior to upgrading. However, you can do this after upgrading too:

fedora-29:~$ sudo dnf install -y elasticsearch-6.6.0-1
Last metadata expiration check: 0:02:17 ago on Sat 30 Mar 2019 12:37:49 PM UTC.
Dependencies resolved.
================================================================================
 Package             Arch         Version         Repository               Size
================================================================================
Installing:
 elasticsearch       noarch       6.6.0-1         elasticsearch-6.x       109 M

Transaction Summary
================================================================================
Install  1 Package

Total download size: 109 M
Installed size: 160 M
Downloading Packages:
elasticsearch-6.6.0.rpm                          59 MB/s | 109 MB     00:01    
--------------------------------------------------------------------------------
Total                                            59 MB/s | 109 MB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Running scriptlet: elasticsearch-6.6.0-1.noarch                           1/1 
Creating elasticsearch group... OK
Creating elasticsearch user... OK
  Installing       : elasticsearch-6.6.0-1.noarch                           1/1 
  Running scriptlet: elasticsearch-6.6.0-1.noarch                           1/1 
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch
  Verifying        : elasticsearch-6.6.0-1.noarch                           1/1 

Installed:
  elasticsearch.noarch 6.6.0-1                                                  

Complete!
fedora-29:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-geoip
-> Downloading ingest-geoip from elastic
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed ingest-geoip
fedora-29:~$ sudo dnf upgrade -y elasticsearch
Last metadata expiration check: 0:02:47 ago on Sat 30 Mar 2019 12:37:49 PM UTC.
Dependencies resolved.
================================================================================
 Package             Arch         Version         Repository               Size
================================================================================
Upgrading:
 elasticsearch       noarch       6.7.0-1         elasticsearch-6.x       142 M

Transaction Summary
================================================================================
Upgrade  1 Package

Total download size: 142 M
Downloading Packages:
elasticsearch-6.7.0.rpm                          64 MB/s | 142 MB     00:02    
--------------------------------------------------------------------------------
Total                                            64 MB/s | 142 MB     00:02     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1 
  Running scriptlet: elasticsearch-6.7.0-1.noarch                           1/1 
  Running scriptlet: elasticsearch-6.7.0-1.noarch                           1/2 
  Upgrading        : elasticsearch-6.7.0-1.noarch                           1/2 
  Running scriptlet: elasticsearch-6.7.0-1.noarch                           1/2 
  Running scriptlet: elasticsearch-6.6.0-1.noarch                           2/2 
  Cleanup          : elasticsearch-6.6.0-1.noarch                           2/2 
  Running scriptlet: elasticsearch-6.6.0-1.noarch                           2/2 
  Running scriptlet: elasticsearch-6.7.0-1.noarch                           2/2 
Created elasticsearch keystore in /etc/elasticsearch
  Running scriptlet: elasticsearch-6.6.0-1.noarch                           2/2 
  Verifying        : elasticsearch-6.7.0-1.noarch                           1/2 
  Verifying        : elasticsearch-6.6.0-1.noarch                           2/2 

Upgraded:
  elasticsearch.noarch 6.7.0-1                                                  

Complete!
fedora-29:~$ sudo ls -al /etc/elasticsearch/ingest-geoip
total 65816
drwxr-x---. 2 root elasticsearch       86 Mar 30 12:40 .
drwxr-s---. 3 root elasticsearch      244 Mar 30 12:40 ..
-rw-rw----. 1 root elasticsearch  6173457 Mar 30 12:40 GeoLite2-ASN.mmdb
-rw-rw----. 1 root elasticsearch 57784030 Mar 30 12:40 GeoLite2-City.mmdb
-rw-rw----. 1 root elasticsearch  3428908 Mar 30 12:40 GeoLite2-Country.mmdb
fedora-29:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-geoip
-> removing [ingest-geoip]...
-> preserving plugin config files [/etc/elasticsearch/ingest-geoip] in case of upgrade; use --purge if not needed
fedora-29:~$ sudo ls -al /etc/elasticsearch/ingest-geoip
total 65816
drwxr-x---. 2 root elasticsearch       86 Mar 30 12:40 .
drwxr-s---. 3 root elasticsearch      244 Mar 30 12:40 ..
-rw-rw----. 1 root elasticsearch  6173457 Mar 30 12:40 GeoLite2-ASN.mmdb
-rw-rw----. 1 root elasticsearch 57784030 Mar 30 12:40 GeoLite2-City.mmdb
-rw-rw----. 1 root elasticsearch  3428908 Mar 30 12:40 GeoLite2-Country.mmdb
fedora-29:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove --purge ingest-geoip
-> removing [ingest-geoip]...
fedora-29:~$ sudo ls -al /etc/elasticsearch/ingest-geoip
ls: cannot access '/etc/elasticsearch/ingest-geoip': No such file or directory
fedora-29:~$

Success! As a last resort, you can manually remove these files too.

uhlhosting:

root@logs:/etc/elasticsearch# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove --purge geoip-ingest -&amp;gt; removing [geoip-ingest]...
ERROR: plugin [geoip-ingest] not found; run 'elasticsearch-plugin list' to get list of installed plugins root@logs:/etc/elasticsearch#

Note that the plugin name is ingest-geoip, not geoip-ingest.

Let me know if you run into any more trouble with this, I would be happy to try to help.

uhlhosting · March 30, 2019, 12:49pm

I will check this first thing Monday! Thanks for reply. The issue is somehow now solved in regards to elasticsearch yet logstash is crashing due to some java failures. I have another thread opened to that issue. Will provide feedback on this matter.

Simon_Shrestha · April 6, 2019, 5:57am

I also did a fresh install of elastic search and got the issue below:

simon@simon:~/Downloads$ sudo dpkg -i elasticsearch-6.7.1.deb
[sudo] password for simon:
Selecting previously unselected package elasticsearch.
(Reading database ... 175469 files and directories currently installed.)
Preparing to unpack elasticsearch-6.7.1.deb ...
/usr/bin/java
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (6.7.1) ...
Setting up elasticsearch (6.7.1) ...
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Created elasticsearch keystore in /etc/elasticsearch
Processing triggers for systemd (237-3ubuntu10.15) ...
Processing triggers for ureadahead (0.100.0-20) ...

simon@simon:~/Downloads$ echo $JAVA_HOME
/usr/lib/jvm/java-11-openjdk-amd64/bin/

vnallamothu · April 8, 2019, 1:48pm

After setting the JAVA_HOME environmental variable to "/usr/java/jdk1.8.0_202-amd64/" , and configuring JAVA_HOME value in /etc/sysconfig/elasticsearch, the warning went off.

uhlhosting · April 25, 2019, 9:46am

It seems the issue is not the plugin anymore, while that was properly removed, I cannot overseem to find whats causing me not to use this in past 1 month almost since latest crazy upgrade that killed kibana/elastaicsearch now followed by a repetitive reboots on logstash, making the whole stack unusable.

Kibana server reports: Kibana server is not ready yet

ElasticSearch reports he cant find JVM home alto is setup on local environment.

and yes finally and last logstash restarts every 11 seconds with this:

[2019-03-26T18:49:46,832][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2019-03-26T18:49:49,632][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::GeoIP", {"database"=>"/etc/logstash/GeoLite2-City.mmdb", "add_tag"=>["GeoIP"], "source"=>"src_ip", "id"=>"ac4250e9e717f894c1c2a3aa01daf2ec8e8de38487c86124af25814b9539fd41"}]=>[{"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2019-03-26T18:49:51,836][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2019-03-26T18:49:54,645][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::GeoIP", {"database"=>"/etc/logstash/GeoLite2-City.mmdb", "add_tag"=>["GeoIP"], "source"=>"src_ip", "id"=>"ac4250e9e717f894c1c2a3aa01daf2ec8e8de38487c86124af25814b9539fd41"}]=>[{"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2019-03-26T18:49:56,840][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2019-03-26T18:49:59,658][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::GeoIP", {"database"=>"/etc/logstash/GeoLite2-City.mmdb", "add_tag"=>["GeoIP"], "source"=>"src_ip", "id"=>"ac4250e9e717f894c1c2a3aa01daf2ec8e8de38487c86124af25814b9539fd41"}]=>[{"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2019-03-26T18:50:01,843][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2019-03-26T18:50:03,239][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2019-03-26T18:50:04,670][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::GeoIP", {"database"=>"/etc/logstash/GeoLite2-City.mmdb", "add_tag"=>["GeoIP"], "source"=>"src_ip", "id"=>"ac4250e9e717f894c1c2a3aa01daf2ec8e8de38487c86124af25814b9539fd41"}]=>[{"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2019-03-26T18:50:06,846][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

jasontedor · April 26, 2019, 3:04am

It seems this is your root problem. Can you offer any output or log messages that would help us help you with this issue?

uhlhosting · April 26, 2019, 10:35am

Can you be more specific what logs would be needed or configs?

Topic		Replies	Views
My Elasticsearch wont start after java update Elasticsearch	5	319	October 27, 2022
Failed to start elastic search Elasticsearch	19	9862	June 19, 2021
Elasticsearch error in running service JAVA error Elasticsearch	16	555	October 13, 2023
Elasticsearch.service: Failed with result 'exit-code' Elasticsearch	8	12175	September 30, 2021
Exception in thread "main" java.lang.RuntimeException: starting java failed with [1] : Failed to start Elasticsearch Elasticsearch	2	5121	June 7, 2020

Elasticsearch could not find Java and now won't start

Related Topics