# Elasticsearch could not find Java and now won't start

**URL:** <https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973>\
**Category:** Elasticsearch\
**Created:** [March 26, 2019, 5:03pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973 "2019-03-26T17:03:24Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![sc1](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sc1](https://discuss.elastic.co/u/sc1)\
**Post date:** [March 26, 2019, 5:03pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/1 "2019-03-26T17:03:24Z")

</div>

Hello,

I recently updated 4 Elasticsearch nodes and I now have an error on 2 nodes, I have looked online about how to resolve it or specify but I can't find anything on the specific problem I am having (perhaps I am searching for the wrong thing).

I am not a Java expert but here is the issue:

```
root@vComputerYacho:/usr/share/elasticsearch/bin# sudo service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-03-26 16:20:38 GMT; 37min ago
     Docs: http://www.elastic.co
  Process: 27459 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 27459 (code=exited, status=1/FAILURE)

Mar 26 16:20:14 vComputerYacho systemd[1]: Started Elasticsearch.
Mar 26 16:20:14 vComputerYacho elasticsearch[27459]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Mar 26 16:20:38 vComputerYacho systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 16:20:38 vComputerYacho systemd[1]: elasticsearch.service: Unit entered failed state.
Mar 26 16:20:38 vComputerYacho systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

```

Of the 4 nodes I updated, 2/3 in one cluster are working fine. 1 which isn't clustered is having this issue. Could someone be as kind to advise on how I can resolve this or explain how an update caused this?

---

<div class="post-metadata">

**Author:** ![uhlhosting](https://avatars.discourse-cdn.com/v4/letter/u/c2a13f/32.png) [@uhlhosting](https://discuss.elastic.co/u/uhlhosting)\
**Post date:** [March 26, 2019, 5:59pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/2 "2019-03-26T17:59:40Z")

</div>

I got the same issue:

```
**●** elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: **failed** (Result: exit-code) since Tue 2019-03-26 18:57:24 CET; 615ms ago
Docs: http://www.elastic.co
Process: 23351 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet **(code=exited, status=1/FAI**
Main PID: 23351 (code=exited, status=1/FAILURE)
Mar 26 18:57:14 logs systemd[1]: Started Elasticsearch.
Mar 26 18:57:14 logs elasticsearch[23351]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Mar 26 18:57:24 logs systemd[1]: **elasticsearch.service: Main process exited, code=exited, status=1/FAILURE**
Mar 26 18:57:24 logs systemd[1]: **elasticsearch.service: Failed with result 'exit-code'.**

```

While upon

```
root@logs:/etc/elasticsearch# echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle/jre
```

---

<div class="post-metadata">

**Author:** ![duckulaj](https://avatars.discourse-cdn.com/v4/letter/d/c5a1d2/32.png) [@duckulaj](https://discuss.elastic.co/u/duckulaj)\
**Post date:** [March 26, 2019, 10:38pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/3 "2019-03-26T22:38:14Z")

</div>

Exactly the same problem here

> Blockquote  
> Mar 26 22:29:05 HP-LINUX-MINT elasticsearch[14306]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA\_HOME

> Blockquote  
> echo $JAVA\_HOME  
> /opt/java/jdk1.8.0\_191

Version 6.7.0 and ingest-plugin 6.7.0

---

<div class="post-metadata">

**Author:** ![muhamedHassan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/muhamedhassan/32/42855_2.png) [@muhamedHassan](https://discuss.elastic.co/u/muhamedHassan)\
**Post date:** [March 27, 2019, 7:48am UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/4 "2019-03-27T07:48:12Z")

</div>

same problem here  
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA\_HOME

---

<div class="post-metadata">

**Author:** ![boxclever38](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boxclever38/32/42610_2.png) [@boxclever38](https://discuss.elastic.co/u/boxclever38)\
**Post date:** [March 27, 2019, 4:35pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/5 "2019-03-27T16:35:13Z")

</div>

This happened to me too, but the JAVA HOME error was a red herring. Check your elasticsearch logs and see if there is a plugin conflict. Mine was the geoip-ingest module after upgrading to 6.7  
Removing the module allowed elasticsearch to start normally.

---

<div class="post-metadata">

**Author:** ![sc1](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sc1](https://discuss.elastic.co/u/sc1)\
**Post date:** [March 27, 2019, 5:18pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/6 "2019-03-27T17:18:48Z")

</div>

Thanks Chris, I did have think it was that and went ahead and uninstalled. But I can see from the latest 6.7 documentation, the plugin is now standard so I'm not even sure how to proceed.

I have troubleshooted though and here is the logs:

```
[2019-03-27T16:59:32,887][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [vComputerYacho] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.7.0.jar:6.7.0]
Caused by: java.lang.RuntimeException: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:78) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.0.jar:6.7.0]
	... 6 more
Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.assertDatabaseExistence(IngestGeoIpPlugin.java:150) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.loadDatabaseReaders(IngestGeoIpPlugin.java:102) ~[?:?]
	at org.elasticsearch.ingest.geoip.IngestGeoIpPlugin.getProcessors(IngestGeoIpPlugin.java:76) ~[?:?]
	at org.elasticsearch.ingest.IngestService.processorFactories(IngestService.java:105) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.ingest.IngestService.<init>(IngestService.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:387) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.7.0.jar:6.7.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.7.0.jar:6.7.0]

```

So yes, it looks like the geoip-ingest module is at fault. I have tried to uninstall it:

```
root@vComputerYacho:/# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-geoip
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
-> removing [ingest-geoip]...
ERROR: plugin [ingest-geoip] not found; run 'elasticsearch-plugin list' to get list of installed plugins

```

When I try list modules to see if it still exists it doesnt return anything so I can't actually tell if it's installed or not.

```
root@vComputerYacho:/# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin list
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
```

---

<div class="post-metadata">

**Author:** ![boxclever38](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boxclever38/32/42610_2.png) [@boxclever38](https://discuss.elastic.co/u/boxclever38)\
**Post date:** [March 27, 2019, 6:00pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/7 "2019-03-27T18:00:55Z")

</div>

I think you're having the same experience as me after uninstalling ingest-geoip. The line that was my clue is:

`Caused by: java.io.IOException: expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]`

The folder **/etc/elasticsearch/ingest-geoip** still exists and elasticsearch doesn't like it. I just renamed the folder rather than deleting it (I'm still a little too green at this to feel comfortable just deleting). After renaming the folder to **ingest-geoip-backup** elasticsearch started A-OK.

Let me know if it works for you.

---

<div class="post-metadata">

**Author:** ![boxclever38](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boxclever38/32/42610_2.png) [@boxclever38](https://discuss.elastic.co/u/boxclever38)\
**Post date:** [March 27, 2019, 6:01pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/8 "2019-03-27T18:01:40Z")

</div>

I'd also be curios to know about the nodes that worked OK. How did they compare?

---

<div class="post-metadata">

**Author:** ![mukai](https://avatars.discourse-cdn.com/v4/letter/m/f08c70/32.png) [@mukai](https://discuss.elastic.co/u/mukai)\
**Post date:** [March 27, 2019, 10:28pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/9 "2019-03-27T22:28:59Z")

</div>

You may need "remove --purge".

> Elasticsearch 6.7.0 checks that there are no leftover geoip database files in the plugin configuration directory because the new module does not use them. Therefore, remove the ingest-geoip plugin prior to upgrading to 6.7.0 with the --purge option to also delete the old database files:  
> **\> ./bin/elasticsearch-plugin remove --purge ingest-geoip**  
> Otherwise you will see the following error message upon startup (assuming /etc/elasticsearch/ingest-geoip as the plugin configuration directory):  
> expected database [GeoLite2-ASN.mmdb] to not exist in [/etc/elasticsearch/ingest-geoip]  
> [https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.7.html#\_literal\_ingest\_geoip\_literal\_and\_literal\_ingest\_user\_agent\_literal\_are\_no\_longer\_distributed\_as\_plugins](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.7.html#_literal_ingest_geoip_literal_and_literal_ingest_user_agent_literal_are_no_longer_distributed_as_plugins)

In my case, I could solve it by executing the command.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 28, 2019, 3:54am UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/10 "2019-03-28T03:54:26Z")

</div>

As @mukai mentions, you do need to run a purge on the `ingest-geoip` plugin.

See [https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.7.html#\_literal\_ingest\_geoip\_literal\_and\_literal\_ingest\_user\_agent\_literal\_are\_no\_longer\_distributed\_as\_plugins](https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-6.7.html#_literal_ingest_geoip_literal_and_literal_ingest_user_agent_literal_are_no_longer_distributed_as_plugins)

---

<div class="post-metadata">

**Author:** ![sc1](https://avatars.discourse-cdn.com/v4/letter/s/a88e4f/32.png) [@sc1](https://discuss.elastic.co/u/sc1)\
**Post date:** [March 28, 2019, 11:48am UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/11 "2019-03-28T11:48:29Z")

</div>

Thanks for the replies @warkolm and @mukai

The purge option didn't actually fix it for me as the documentation suggested. When I ran the purge function, because I had previously performed the remove function, this was the result:

```
root@vComputerYacho:~# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove --purge geoip-ingest
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
-> removing [geoip-ingest]...
ERROR: plugin [geoip-ingest] not found; run 'elasticsearch-plugin list' to get list of installed plugins   

```

I checked the logs and could see the issue was in /etc/elasticsearch/ingest-geoip/

`rm -rf /etc/elasticsearch/ingest-geoip/`

After performing that, my 1st node was working again. I have also replicated on the other node, which I hadn't troubleshoot. The purge option alone is not enough, you still have to `rm -rf` the folder in the`/etc/elasticsearch`

Thanks again for the help!

---

<div class="post-metadata">

**Author:** ![uhlhosting](https://avatars.discourse-cdn.com/v4/letter/u/c2a13f/32.png) [@uhlhosting](https://discuss.elastic.co/u/uhlhosting)\
**Post date:** [March 29, 2019, 9:25pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/12 "2019-03-29T21:25:50Z")

</div>

sasd

---

<div class="post-metadata">

**Author:** ![uhlhosting](https://avatars.discourse-cdn.com/v4/letter/u/c2a13f/32.png) [@uhlhosting](https://discuss.elastic.co/u/uhlhosting)\
**Post date:** [March 29, 2019, 9:28pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/13 "2019-03-29T21:28:41Z")

</div>

The option to remove ingest plugin is not working since the plugin list shows none installed.  
Yet I get this:

The command to install a plugin then throws this:  
ERROR: Unknown plugin geoip-ingest

Anyone any ideas?

I wonder how people with OSS managed to make $ from so much ignorance.

```
root@logs:/etc/elasticsearch# locate ingest-geoip
/etc/elasticsearch/ingest-geoip
/usr/share/elasticsearch/modules/ingest-geoip
/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-ASN.mmdb
/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-City.mmdb
/usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-Country.mmdb
/usr/share/elasticsearch/modules/ingest-geoip/geoip2-2.9.0.jar
/usr/share/elasticsearch/modules/ingest-geoip/ingest-geoip-6.7.0.jar
/usr/share/elasticsearch/modules/ingest-geoip/jackson-annotations-2.8.11.jar
/usr/share/elasticsearch/modules/ingest-geoip/jackson-databind-2.8.11.3.jar
/usr/share/elasticsearch/modules/ingest-geoip/maxmind-db-1.2.2.jar
/usr/share/elasticsearch/modules/ingest-geoip/plugin-descriptor.properties
/usr/share/elasticsearch/modules/ingest-geoip/plugin-security.policy
root@logs:/etc/elasticsearch# rm -rf /etc/elasticsearch/ingest-geoip
root@logs:/etc/elasticsearch# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove --purge geoip-ingest
-&gt; removing [geoip-ingest]...
ERROR: plugin [geoip-ingest] not found; run 'elasticsearch-plugin list' to get list of installed plugins
root@logs:/etc/elasticsearch#

```

And from the logs:  
[https://pastebin.com/DYdj90iE](https://pastebin.com/DYdj90iE)

After a while decided to manually remove all plugins and modules, and reinstalled elasticsearch, this helped, while meanwhile, logstash fails with:

```
[2019-03-29T23:02:47,796][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:03:07,668][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:03:28,046][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:03:47,884][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:04:08,362][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:04:28,204][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:04:48,694][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-03-29T23:05:08,807][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
```

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [March 30, 2019, 12:46pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/14 "2019-03-30T12:46:55Z")

</div>

Sorry that you're having trouble.

As the docs say:

```auto
Therefore, remove the ingest-geoip plugin prior to upgrading to 6.7.0 with the --purge option to also delete the old database files

```

Note that this should be done _prior_ to upgrading. However, you can do this _after_ upgrading too:

```auto
fedora-29:~$ sudo dnf install -y elasticsearch-6.6.0-1
Last metadata expiration check: 0:02:17 ago on Sat 30 Mar 2019 12:37:49 PM UTC.
Dependencies resolved.
================================================================================
 Package Arch Version Repository Size
================================================================================
Installing:
 elasticsearch noarch 6.6.0-1 elasticsearch-6.x 109 M

Transaction Summary
================================================================================
Install 1 Package

Total download size: 109 M
Installed size: 160 M
Downloading Packages:
elasticsearch-6.6.0.rpm 59 MB/s | 109 MB 00:01    
--------------------------------------------------------------------------------
Total 59 MB/s | 109 MB 00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing : 1/1 
  Running scriptlet: elasticsearch-6.6.0-1.noarch 1/1 
Creating elasticsearch group... OK
Creating elasticsearch user... OK
  Installing : elasticsearch-6.6.0-1.noarch 1/1 
  Running scriptlet: elasticsearch-6.6.0-1.noarch 1/1 
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service
Created elasticsearch keystore in /etc/elasticsearch
  Verifying : elasticsearch-6.6.0-1.noarch 1/1 

Installed:
  elasticsearch.noarch 6.6.0-1                                                  

Complete!
fedora-29:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch ingest-geoip
-> Downloading ingest-geoip from elastic
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed ingest-geoip
fedora-29:~$ sudo dnf upgrade -y elasticsearch
Last metadata expiration check: 0:02:47 ago on Sat 30 Mar 2019 12:37:49 PM UTC.
Dependencies resolved.
================================================================================
 Package Arch Version Repository Size
================================================================================
Upgrading:
 elasticsearch noarch 6.7.0-1 elasticsearch-6.x 142 M

Transaction Summary
================================================================================
Upgrade 1 Package

Total download size: 142 M
Downloading Packages:
elasticsearch-6.7.0.rpm 64 MB/s | 142 MB 00:02    
--------------------------------------------------------------------------------
Total 64 MB/s | 142 MB 00:02     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing : 1/1 
  Running scriptlet: elasticsearch-6.7.0-1.noarch 1/1 
  Running scriptlet: elasticsearch-6.7.0-1.noarch 1/2 
  Upgrading : elasticsearch-6.7.0-1.noarch 1/2 
  Running scriptlet: elasticsearch-6.7.0-1.noarch 1/2 
  Running scriptlet: elasticsearch-6.6.0-1.noarch 2/2 
  Cleanup : elasticsearch-6.6.0-1.noarch 2/2 
  Running scriptlet: elasticsearch-6.6.0-1.noarch 2/2 
  Running scriptlet: elasticsearch-6.7.0-1.noarch 2/2 
Created elasticsearch keystore in /etc/elasticsearch
  Running scriptlet: elasticsearch-6.6.0-1.noarch 2/2 
  Verifying : elasticsearch-6.7.0-1.noarch 1/2 
  Verifying : elasticsearch-6.6.0-1.noarch 2/2 

Upgraded:
  elasticsearch.noarch 6.7.0-1                                                  

Complete!
fedora-29:~$ sudo ls -al /etc/elasticsearch/ingest-geoip
total 65816
drwxr-x---. 2 root elasticsearch 86 Mar 30 12:40 .
drwxr-s---. 3 root elasticsearch 244 Mar 30 12:40 ..
-rw-rw----. 1 root elasticsearch 6173457 Mar 30 12:40 GeoLite2-ASN.mmdb
-rw-rw----. 1 root elasticsearch 57784030 Mar 30 12:40 GeoLite2-City.mmdb
-rw-rw----. 1 root elasticsearch 3428908 Mar 30 12:40 GeoLite2-Country.mmdb
fedora-29:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove ingest-geoip
-> removing [ingest-geoip]...
-> preserving plugin config files [/etc/elasticsearch/ingest-geoip] in case of upgrade; use --purge if not needed
fedora-29:~$ sudo ls -al /etc/elasticsearch/ingest-geoip
total 65816
drwxr-x---. 2 root elasticsearch 86 Mar 30 12:40 .
drwxr-s---. 3 root elasticsearch 244 Mar 30 12:40 ..
-rw-rw----. 1 root elasticsearch 6173457 Mar 30 12:40 GeoLite2-ASN.mmdb
-rw-rw----. 1 root elasticsearch 57784030 Mar 30 12:40 GeoLite2-City.mmdb
-rw-rw----. 1 root elasticsearch 3428908 Mar 30 12:40 GeoLite2-Country.mmdb
fedora-29:~$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove --purge ingest-geoip
-> removing [ingest-geoip]...
fedora-29:~$ sudo ls -al /etc/elasticsearch/ingest-geoip
ls: cannot access '/etc/elasticsearch/ingest-geoip': No such file or directory
fedora-29:~$ 

```

Success! As a last resort, you can manually remove these files too.

> [@uhlhosting](#):
>
> ```auto
> root@logs:/etc/elasticsearch# sudo /usr/share/elasticsearch/bin/elasticsearch-plugin remove --purge geoip-ingest -&amp;gt; removing [geoip-ingest]...
> ERROR: plugin [geoip-ingest] not found; run 'elasticsearch-plugin list' to get list of installed plugins root@logs:/etc/elasticsearch#
> 
> ```

Note that the plugin name is `ingest-geoip`, not `geoip-ingest`.

Let me know if you run into any more trouble with this, I would be happy to try to help.

---

<div class="post-metadata">

**Author:** ![uhlhosting](https://avatars.discourse-cdn.com/v4/letter/u/c2a13f/32.png) [@uhlhosting](https://discuss.elastic.co/u/uhlhosting)\
**Post date:** [March 30, 2019, 12:49pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/15 "2019-03-30T12:49:54Z")

</div>

I will check this first thing Monday! Thanks for reply. The issue is somehow now solved in regards to elasticsearch yet logstash is crashing due to some java failures. I have another thread opened to that issue. Will provide feedback on this matter.

---

<div class="post-metadata">

**Author:** ![Simon\_Shrestha](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simon_shrestha/32/43588_2.png) [@Simon\_Shrestha](https://discuss.elastic.co/u/Simon_Shrestha)\
**Post date:** [April 6, 2019, 5:57am UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/16 "2019-04-06T05:57:19Z")

</div>

I also did a fresh install of elastic search and got the issue below:

simon@simon:~/Downloads$ sudo dpkg -i elasticsearch-6.7.1.deb  
[sudo] password for simon:  
Selecting previously unselected package elasticsearch.  
(Reading database ... 175469 files and directories currently installed.)  
Preparing to unpack elasticsearch-6.7.1.deb ...  
/usr/bin/java  
Creating elasticsearch group... OK  
Creating elasticsearch user... OK  
Unpacking elasticsearch (6.7.1) ...  
Setting up elasticsearch (6.7.1) ...  
warning: Falling back to java on path. This behavior is deprecated. Specify JAVA\_HOME  
Created elasticsearch keystore in /etc/elasticsearch  
Processing triggers for systemd (237-3ubuntu10.15) ...  
Processing triggers for ureadahead (0.100.0-20) ...

simon@simon:~/Downloads$ echo $JAVA\_HOME  
/usr/lib/jvm/java-11-openjdk-amd64/bin/

---

<div class="post-metadata">

**Author:** ![vnallamothu](https://avatars.discourse-cdn.com/v4/letter/v/35a633/32.png) [@vnallamothu](https://discuss.elastic.co/u/vnallamothu)\
**Post date:** [April 8, 2019, 1:48pm UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/17 "2019-04-08T13:48:56Z")

</div>

After setting the JAVA\_HOME environmental variable to "/usr/java/jdk1.8.0\_202-amd64/" , and configuring JAVA\_HOME value in /etc/sysconfig/elasticsearch, the warning went off.

---

<div class="post-metadata">

**Author:** ![uhlhosting](https://avatars.discourse-cdn.com/v4/letter/u/c2a13f/32.png) [@uhlhosting](https://discuss.elastic.co/u/uhlhosting)\
**Post date:** [April 25, 2019, 9:46am UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/18 "2019-04-25T09:46:02Z")

</div>

It seems the issue is not the plugin anymore, while that was properly removed, I cannot overseem to find whats causing me not to use this in past 1 month almost since latest crazy upgrade that killed kibana/elastaicsearch now followed by a repetitive reboots on logstash, making the whole stack unusable.

Kibana server reports: Kibana server is not ready yet

ElasticSearch reports he cant find JVM home alto is setup on local environment.

and yes finally and last logstash restarts every 11 seconds with this:

```
[2019-03-26T18:49:46,832][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2019-03-26T18:49:49,632][WARN][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::GeoIP", {"database"=>"/etc/logstash/GeoLite2-City.mmdb", "add_tag"=>["GeoIP"], "source"=>"src_ip", "id"=>"ac4250e9e717f894c1c2a3aa01daf2ec8e8de38487c86124af25814b9539fd41"}]=>[{"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2019-03-26T18:49:51,836][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2019-03-26T18:49:54,645][WARN][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::GeoIP", {"database"=>"/etc/logstash/GeoLite2-City.mmdb", "add_tag"=>["GeoIP"], "source"=>"src_ip", "id"=>"ac4250e9e717f894c1c2a3aa01daf2ec8e8de38487c86124af25814b9539fd41"}]=>[{"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2019-03-26T18:49:56,840][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2019-03-26T18:49:59,658][WARN][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::GeoIP", {"database"=>"/etc/logstash/GeoLite2-City.mmdb", "add_tag"=>["GeoIP"], "source"=>"src_ip", "id"=>"ac4250e9e717f894c1c2a3aa01daf2ec8e8de38487c86124af25814b9539fd41"}]=>[{"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2019-03-26T18:50:01,843][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2019-03-26T18:50:03,239][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
[2019-03-26T18:50:04,670][WARN][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{["LogStash::Filters::GeoIP", {"database"=>"/etc/logstash/GeoLite2-City.mmdb", "add_tag"=>["GeoIP"], "source"=>"src_ip", "id"=>"ac4250e9e717f894c1c2a3aa01daf2ec8e8de38487c86124af25814b9539fd41"}]=>[{"thread_id"=>32, "name"=>nil, "current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
[2019-03-26T18:50:06,846][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://172.18.111.1:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://172.18.111.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
```

---

<div class="post-metadata">

**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\
**Post date:** [April 26, 2019, 3:04am UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/19 "2019-04-26T03:04:27Z")

</div>

> [@uhlhosting](#):
>
> Elasticsearch reports he cant find JVM home alto is setup on local environment.

It seems this is your root problem. Can you offer any output or log messages that would help us help you with this issue?

---

<div class="post-metadata">

**Author:** ![uhlhosting](https://avatars.discourse-cdn.com/v4/letter/u/c2a13f/32.png) [@uhlhosting](https://discuss.elastic.co/u/uhlhosting)\
**Post date:** [April 26, 2019, 10:35am UTC](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973/20 "2019-04-26T10:35:48Z")

</div>

Can you be more specific what logs would be needed or configs?

[Next page](https://discuss.elastic.co/t/elasticsearch-could-not-find-java-and-now-wont-start/173973.md?page=2)