Elasticsearch 8.1.1 error

Elastic Stack Elasticsearch
1

I am trying to upgrade from 7.17.1 to 8.1.1 and getting the following error on RHEL 7.x latest with Java 17 from RHEL:

[2022-03-30T11:51:49,898][INFO ][o.e.n.Node               ] [elasticsearch01.internaldomain.com] version[8.1.1], pid[39237], build[default/tar/d0925dd6f22e07b935750420a3155db6e5c58381/2022-03-17T22:01:32.658689558Z], OS[Linux/3.10.0-1160.53.1.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.2/17.0.2+8]
[2022-03-30T11:51:49,903][INFO ][o.e.n.Node               ] [elasticsearch01.internaldomain.com] JVM home [/opt/elastic/elasticsearch-8.1.1/jdk], using bundled JDK [true]
[2022-03-30T11:51:49,904][INFO ][o.e.n.Node               ] [elasticsearch01.internaldomain.com] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms2g, -Xmx2g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-5639709805910131003, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=1, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -Des.path.home=/opt/elastic/current, -Des.path.conf=/opt/elastic/current/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-03-30T11:51:53,096][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [aggs-matrix-stats]
[2022-03-30T11:51:53,096][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [analysis-common]
[2022-03-30T11:51:53,097][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [constant-keyword]
[2022-03-30T11:51:53,097][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [data-streams]
[2022-03-30T11:51:53,097][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [frozen-indices]
[2022-03-30T11:51:53,097][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [ingest-common]
[2022-03-30T11:51:53,098][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [ingest-geoip]
[2022-03-30T11:51:53,098][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [ingest-user-agent]
[2022-03-30T11:51:53,098][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [kibana]
[2022-03-30T11:51:53,099][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [lang-expression]
[2022-03-30T11:51:53,099][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [lang-mustache]
[2022-03-30T11:51:53,099][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [lang-painless]
[2022-03-30T11:51:53,099][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [legacy-geo]
[2022-03-30T11:51:53,100][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [mapper-extras]
[2022-03-30T11:51:53,100][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [mapper-version]
[2022-03-30T11:51:53,100][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [old-lucene-versions]
[2022-03-30T11:51:53,100][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [parent-join]
[2022-03-30T11:51:53,101][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [percolator]
[2022-03-30T11:51:53,101][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [rank-eval]
[2022-03-30T11:51:53,101][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [reindex]
[2022-03-30T11:51:53,101][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [repositories-metering-api]
[2022-03-30T11:51:53,102][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [repository-azure]
[2022-03-30T11:51:53,102][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [repository-encrypted]
[2022-03-30T11:51:53,102][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [repository-gcs]
[2022-03-30T11:51:53,102][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [repository-s3]
[2022-03-30T11:51:53,103][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [repository-url]
[2022-03-30T11:51:53,103][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [runtime-fields-common]
[2022-03-30T11:51:53,103][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [search-business-rules]
[2022-03-30T11:51:53,103][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [searchable-snapshots]
[2022-03-30T11:51:53,103][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [snapshot-based-recoveries]
[2022-03-30T11:51:53,104][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [snapshot-repo-test-kit]
[2022-03-30T11:51:53,104][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [spatial]
[2022-03-30T11:51:53,104][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [transform]
[2022-03-30T11:51:53,104][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [transport-netty4]
[2022-03-30T11:51:53,105][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [unsigned-long]
[2022-03-30T11:51:53,105][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [vector-tile]
[2022-03-30T11:51:53,105][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [vectors]
[2022-03-30T11:51:53,105][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [wildcard]
[2022-03-30T11:51:53,106][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-aggregate-metric]
[2022-03-30T11:51:53,106][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-analytics]
[2022-03-30T11:51:53,106][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-async]
[2022-03-30T11:51:53,106][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-async-search]
[2022-03-30T11:51:53,107][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-autoscaling]
[2022-03-30T11:51:53,107][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-ccr]
[2022-03-30T11:51:53,107][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-core]
[2022-03-30T11:51:53,107][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-deprecation]
[2022-03-30T11:51:53,108][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-enrich]
[2022-03-30T11:51:53,108][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-eql]
[2022-03-30T11:51:53,108][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-fleet]
[2022-03-30T11:51:53,108][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-graph]
[2022-03-30T11:51:53,108][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-identity-provider]
[2022-03-30T11:51:53,109][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-ilm]
[2022-03-30T11:51:53,109][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-logstash]
[2022-03-30T11:51:53,109][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-ml]
[2022-03-30T11:51:53,109][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-monitoring]
[2022-03-30T11:51:53,110][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-ql]
[2022-03-30T11:51:53,110][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-rollup]
[2022-03-30T11:51:53,110][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-security]
[2022-03-30T11:51:53,110][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-shutdown]
[2022-03-30T11:51:53,110][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-sql]
[2022-03-30T11:51:53,111][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-stack]
[2022-03-30T11:51:53,111][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-text-structure]
[2022-03-30T11:51:53,111][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-voting-only-node]
[2022-03-30T11:51:53,111][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] loaded module [x-pack-watcher]
[2022-03-30T11:51:53,112][INFO ][o.e.p.PluginsService     ] [elasticsearch01.internaldomain.com] no plugins loaded
[2022-03-30T11:51:53,209][ERROR][o.e.b.Bootstrap          ] [elasticsearch01.internaldomain.com] Exception
org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.node.Node.<init>(Node.java:1023) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:284) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) [elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) [elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-8.1.1.jar:8.1.1]
        at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-8.1.1.jar:8.1.1]
Caused by: java.nio.file.FileAlreadyExistsException: /opt/elastic/current
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:94) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397) ~[?:?]
        at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]
        at java.nio.file.Files.createDirectories(Files.java:753) ~[?:?]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:261) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:397) ~[elasticsearch-8.1.1.jar:8.1.1]
        ... 11 more
[2022-03-30T11:51:53,217][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch01.internaldomain.com] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:81) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-8.1.1.jar:8.1.1]
        at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-8.1.1.jar:8.1.1]
Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.node.Node.<init>(Node.java:1023) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:284) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-8.1.1.jar:8.1.1]
        ... 6 more
Caused by: java.nio.file.FileAlreadyExistsException: /opt/elastic/current
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:94) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397) ~[?:?]
        at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]
        at java.nio.file.Files.createDirectories(Files.java:753) ~[?:?]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:261) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:397) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.node.Node.<init>(Node.java:284) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:358) ~[elasticsearch-8.1.1.jar:8.1.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-8.1.1.jar:8.1.1]
        ... 6 more
2

Welcome to our community! :smiley:

Can you share your elasticsearch.yml?

3 
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /opt/elastic/current/
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
action.auto_create_index: true
network.host: 0.0.0.0
path.repo: ["/opt/elastic/backup/"]
#enable_snapshot_restore_privilege: true
#node.max_local_storage_nodes=2
discovery.type: single-node
4

Any reason you are using a custom path.data?
Is there anything in there? Does it have the right permissions for Elasticsearch to access it?

5

This was deployed custom. Downloaded the .tar.gz and ran it there. If I didn't have the path.data in there would it just use the directory I installed it in?

Elasticsearch does have full rights to the folder.

6

Any help?

7

Yes it would.

I'm not sure why this would be happening sorry. @DavidTurner would you have a better idea mate?

8

Not sure, but this message suggests to me that this is a file and not a directory. Maybe it's a symlink? Try using an actual directory instead.

1 Like
9

You are correct! However I have been using symlinks in prior releases. Did something change?

10

I think it might be this change - prior to 8.0 we would have been trying to create a subdirectory of path.data which would have been ok with a symlink in its ancestry, but now we try and create path.data itself which apparently fails on a symlink.

I opened `path.data` can no longer be a symlink · Issue #85701 · elastic/elasticsearch · GitHub to report this.

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.