Elasticsearch Service won't start

Jensxy · July 23, 2020, 9:28am

I did sudo chmod -R 750 /var/log/elasticsearch/ and now I get the following messages

/usr/share/elasticsearch$ sudo -u elasticsearch ./bin/elasticsearch
future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk/jdk1.8.0_202/jre] does not meet this requirement
future versions of Elasticsearch will require Java 11; your Java version from [/opt/jdk/jdk1.8.0_202/jre] does not meet this requirement
[2020-07-23T11:26:56,232][INFO ][o.e.n.Node               ] [SERVER1] version[7.8.0], pid[28095], build[default/deb/757314695644ea9a1dc2fecd26d1a43856725e65/2020-06-14T19:35:50.234439Z], OS[Linux/4.15.0-108-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_202/25.202-b08]
[2020-07-23T11:26:56,233][INFO ][o.e.n.Node               ] [SERVER1] JVM home [/opt/jdk/jdk1.8.0_202/jre]
[2020-07-23T11:26:56,233][INFO ][o.e.n.Node               ] [SERVER1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,JRE, -Xms13g, -Xmx13g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-6288221564287275426, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -XX:MaxDirectMemorySize=6979321856, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2020-07-23T11:26:59,128][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [SERVER1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.5.0 but version 7.8.0 is running
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.8.0.jar:7.8.0]
Caused by: java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.5.0 but version 7.8.0 is running
        at org.elasticsearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:348) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:533) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.node.Node.<init>(Node.java:317) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
        ... 6 more
uncaught exception in thread [main]
java.lang.IllegalArgumentException: Plugin [ingest-attachment] was built for Elasticsearch version 7.5.0 but version 7.8.0 is running
        at org.elasticsearch.plugins.PluginsService.verifyCompatibility(PluginsService.java:348)
        at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:533)
        at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:473)
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:164)
        at org.elasticsearch.node.Node.<init>(Node.java:317)
        at org.elasticsearch.node.Node.<init>(Node.java:266)
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227)
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
        at org.elasticsearch.cli.Command.main(Command.java:90)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log

Is the problem maybe due to the ingest-attachment plugin?

EDIT: Reinstalling the ingest-attachment plugin solved the problem. https://discuss.elastic.co/t/elastic-search-crashes-after-start/169957/3

Steve_Mushero · July 23, 2020, 9:45am

Getting some where - note your ./bin/elasticsearch is using the OS level Java, not the bundled V14 one - why? Should try to fix that, but yes, the current issue is the plugin - then what happened after you 'solved' that?

Jensxy · July 23, 2020, 9:52am

I just started the service and the status shows

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-07-23 11:34:10 CEST; 15min ago
     Docs: https://www.elastic.co
 Main PID: 29428 (java)
    Tasks: 89 (limit: 4915)
   CGroup: /system.slice/elasticsearch.service
           ├─29428 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -
           └─29643 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Jul 23 11:33:52 SERVER1 systemd[1]: Starting Elasticsearch...
Jul 23 11:33:54 SERVER1 systemd-entrypoint[29428]: OpenJDK 64-Bit Server VM warning: Ignoring option UseConcMarkSweepGC; support was removed in 14.0
Jul 23 11:33:54 SERVER1 systemd-entrypoint[29428]: OpenJDK 64-Bit Server VM warning: Ignoring option CMSInitiatingOccupancyFraction; support was removed in 14.0
Jul 23 11:33:54 SERVER1 systemd-entrypoint[29428]: OpenJDK 64-Bit Server VM warning: Ignoring option UseCMSInitiatingOccupancyOnly; support was removed in 14.0

It is using the correct Java now, isn't it? And I don't know why it used the OS level Java.

Steve_Mushero · July 24, 2020, 1:11am

Yes, seems good now; just make sure it's repeatable and things start correctly when the VM boots, etc. Was an odd chain of events and messed up log permissions, etc. so all kinda weird.

Anyway, good luck.

Vinayak_Sapre · July 24, 2020, 1:44am

@Jensxy
Glad that finally you were able start the service.

Here is a snippet from jvm.options bundled with ES 7.8.0. Notice jdk 14 specific settings and UseConcMarkSweepGC marked to be used with JDK 8 through 13 only.

## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30

With this you should not see warning messages like

Jul 23 11:33:54 SERVER1 systemd-entrypoint[29428]: OpenJDK 64-Bit Server VM warning: Ignoring option UseConcMarkSweepGC; support was removed in 14.0

JDK14 by default uses +UserG1GC. Not sure if other two settings have performance impact.

Steve_Mushero · July 24, 2020, 8:06am

I wonder if this node has always been like this, with left over init / start scripts from old versions, etc. Maybe get new one from clean install and copy over, making sure the right JVM is being used (and/or remove the OS-supplied JVM to make sure).

system · August 21, 2020, 8:06am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.