Error encountered after modified elasticsearch.yml

Elastic Stack Elasticsearch
1

Hi,

I am trying out new version Elasticsearch 6.2.3 on AWS Ubuntu 16.04
After configuring the elasticsearch.yml file I received these error
ubuntu@ip-172-31-25-205:/etc$ sudo /etc/init.d/elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-03-26 01:49:47 UTC; 3s ago
Docs: http://www.elastic.co
Process: 8060 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 8060 (code=exited, status=1/FAILURE)

Mar 26 01:49:41 ip-172-31-25-205 systemd[1]: Started Elasticsearch.
Mar 26 01:49:47 ip-172-31-25-205 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 01:49:47 ip-172-31-25-205 systemd[1]: elasticsearch.service: Unit entered failed state.
Mar 26 01:49:47 ip-172-31-25-205 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Here's the log details

root@ip-172-31-25-205:/var/log/elasticsearch# more yk_cluster.log
[2018-03-26T01:11:34,113][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.2.jar:6.2.2]
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_151]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_151]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_151]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_151]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_151]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_151]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_151]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:421) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.2.2.jar:6.2.2]
... 12 more
[2018-03-26T01:11:34,145][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [yk_search1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.2.jar:6.2.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.2.jar:6.2.2]
Caused by: java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.2.jar:6.2.2]
... 6 more

2

continued log due to word limits

Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_151]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_151]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_151]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_151]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_151]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_151]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_151]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:421) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:68) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.2.jar:6.2.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.2.jar:6.2.2]
... 6 more
[2018-03-26T01:49:43,188][INFO ][o.e.n.Node ] [yk_search1] initializing ...
[2018-03-26T01:49:43,304][INFO ][o.e.e.NodeEnvironment ] [yk_search1] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [27.6gb], net total_space [29gb], types [ext4]
[2018-03-26T01:49:43,305][INFO ][o.e.e.NodeEnvironment ] [yk_search1] heap size [491.6mb], compressed ordinary object pointers [true]
[2018-03-26T01:49:43,306][INFO ][o.e.n.Node ] [yk_search1] node name [yk_search1], node ID [LlNC66V1Tr-5ygQzoaq7DQ]
[2018-03-26T01:49:43,309][INFO ][o.e.n.Node ] [yk_search1] version[6.2.2], pid[8060], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.4.0-1052-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.1
51-b12]
[2018-03-26T01:49:43,310][INFO ][o.e.n.Node ] [yk_search1] JVM arguments [-Xms500m, -Xmx500m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -
Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=fals
e, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.Rd5vHzo3, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintG
CApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch]
[2018-03-26T01:49:44,474][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [aggs-matrix-stats]
[2018-03-26T01:49:44,474][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [analysis-common]
[2018-03-26T01:49:44,475][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [ingest-common]
[2018-03-26T01:49:44,475][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [lang-expression]
[2018-03-26T01:49:44,475][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [lang-mustache]
[2018-03-26T01:49:44,475][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [lang-painless]
[2018-03-26T01:49:44,475][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [mapper-extras]
[2018-03-26T01:49:44,475][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [parent-join]
[2018-03-26T01:49:44,475][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [percolator]
[2018-03-26T01:49:44,476][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [rank-eval]
[2018-03-26T01:49:44,476][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [reindex]
[2018-03-26T01:49:44,476][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [repository-url]
[2018-03-26T01:49:44,476][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [transport-netty4]
[2018-03-26T01:49:44,476][INFO ][o.e.p.PluginsService ] [yk_search1] loaded module [tribe]
[2018-03-26T01:49:44,476][INFO ][o.e.p.PluginsService ] [yk_search1] no plugins loaded
[2018-03-26T01:49:47,696][WARN ][o.e.c.s.SettingsModule ] [yk_search1]

Found index level settings on node level configuration.

Since elasticsearch 5.x index level settings can NOT be set on the nodes
configuration like the elasticsearch.yaml, in system properties or command line
arguments.In order to upgrade all indices the settings must be updated via the
/${index}/_settings API. Unless all settings are dynamic all indices must be closed
in order to apply the upgradeIndices created in the future should use index templates
to set default values.

Please ensure all required values are updated on all indices by executing:

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
"index.number_of_replicas" : "0",
"index.number_of_shards" : "5"
}'

3

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

4

Hello,

My apologise as it was my first post and not to clear on the format..please be patience while I try to make it easier to read here.

I am trying out new version Elasticsearch 6.2.3 on AWS Ubuntu 16.04
After configuring the elasticsearch.yml file to setup the cluster & node name, I received the following error message

ubuntu@ip-172-31-25-205:/etc$ sudo /etc/init.d/elasticsearch status
● elasticsearch.service - Elastic-search
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-03-26 01:49:47 UTC; 3s ago
Docs: http://www.elastic.co
Process: 8060 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 8060 (code=exited, status=1/FAILURE)

Can help on what is wrong here. Thanks.

5

This recent log is correctly formatted thanks. Could you update your previous posts and do the same? That will help for clarity.

Please share your elasticsearch.yml file.

Here I guess that you set in it number_of_replicas and number_of_shards which is not allowed anymore as you can see in your logs.

6

not access ? change to es:es about the data directory /usr/share/elasticsearch/data

7

No that's an old trace. See the timestamp [2018-03-26T01:11:34,113]

Then he restarted at [2018-03-26T01:49:43,188].

8

Here's the elasticsearch.yml file, yes i had set the number_of_replicas and number_of_shards which I just put in the comment below, unfortunately the error still persist.

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: yk_cluster
#index.number_of_shards: 5
#index.number_of_replicas: 0
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: yk_search1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6)
#
network.host: 172.31.25.205
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["172.31.25.205", "172.31.23.156"]
discovery.zen.ping.multicast.enabled: false

# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
9

If the error message is exactly the same (please share it), that means that the configuration file read by elasticsearch is not the one you modified.

BTW, here is a simpler version more readable:

cluster.name: yk_cluster
node.name: yk_search1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.31.25.205
discovery.zen.ping.unicast.hosts: ["172.31.25.205", "172.31.23.156"]
discovery.zen.ping.multicast.enabled: false

Note that you can probably remove discovery.zen.ping.multicast.enabled: false. Multicast is not available anymore.

10

These is the error message

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2018-03-26 06:57:36 UTC; 18h ago
     Docs: http://www.elastic.co
  Process: 9103 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 9103 (code=exited, status=1/FAILURE)

Mar 26 06:57:29 ip-172-31-25-205 systemd[1]: Started Elasticsearch.
Mar 26 06:57:36 ip-172-31-25-205 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 06:57:36 ip-172-31-25-205 systemd[1]: elasticsearch.service: Unit entered failed state.
Mar 26 06:57:36 ip-172-31-25-205 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
11

What is the error message in elasticsearch logs?

12

Here's the error log

root@ip-172-31-25-205:/var/log/elasticsearch# vi yk_cluster.log

    [2018-03-26T01:11:34,113][ERROR][o.e.b.Bootstrap          ] Exception
    java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data)
            at org.elasticsearch.bootstrap.FilePermissionUtils.addDirectoryPath(FilePermissionUtils.java:70) ~[elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:307) ~[elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:262) ~[elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.bootstrap.Security.configure(Security.java:123) ~[elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:208) ~[elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) [elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) [elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) [elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.2.2.jar:6.2.2]
            at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.2.2.jar:6.2.2]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.2.2.jar:6.2.2]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.2.2.jar:6.2.2]
    Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data
13

You don't have access to

/usr/share/elasticsearch/data

Check that the user running elasticsearch has all the rights.

14

I tried login to root but can't find the folder /usr/share/elasticsearch/data

root@ip-172-31-25-205:/usr/share/elasticsearch# ls -ltr
total 216
-rw-r--r--  1 root root   9267 Feb 16 19:03 README.textile
drwxr-xr-x  2 root root   4096 Feb 16 19:03 plugins
-rw-r--r--  1 root root 191887 Feb 16 19:03 NOTICE.txt
drwxr-xr-x 16 root root   4096 Mar 20 00:45 modules
drwxr-xr-x  2 root root   4096 Mar 20 00:45 lib
drwxr-xr-x  2 root root   4096 Mar 20 00:45 bin
15

I think that the problem here is that everything is under the root user.

I don't think you followed the installation guide, did you?

16

Yes, I followed some online training guide and probably the version is older.

I tried to search for the official installation guide in elastic site for the current version but cannot find it. Any clue where to find?

Thanks

17

I'd start here: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html

18

Thanks for the setup guide.

Before I proceed will need to uninstall first, here's the steps which I got from one of the forum member advise, is it ok to proceed with this?

Find where is elasticsearch:

find / --name "elasticsearch"

Then remove the folders. Also, remove service script at

/etc/init.d/elasticsearch
19

I believe it's correct.

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.