Elastic Search not update config (elasticsearch.yml)

Elastic Stack Elasticsearch
1

I changed the configuration to elasticsearch.yml

Current part of the configuration:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /es_certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /es_certs/elastic-certificates.p12

After rebooting elasticsearch I get:

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2019-11-17 08:43:27 UTC; 2min 37s ago
Docs: http://www.elastic.co
Process: 2970 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 2970 (code=exited, status=1/FAILURE)

Logs:

Nov 17 06:25:02 icluster-node-d000 rsyslogd: [origin software="rsyslogd" swVersion="8.32.0" x-pid="1011" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 17 06:47:01 icluster-node-d000 CRON[2808]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly ))
Nov 17 06:53:16 icluster-node-d000 systemd-timesyncd[678]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Nov 17 06:53:16 icluster-node-d000 systemd-timesyncd[678]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Nov 17 07:17:01 icluster-node-d000 CRON[2823]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 17 07:29:11 icluster-node-d000 systemd-timesyncd[678]: Timed out waiting for reply from 91.189.89.198:123 (ntp.ubuntu.com).
Nov 17 08:03:31 icluster-node-d000 systemd-timesyncd[678]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Nov 17 08:03:41 icluster-node-d000 systemd-timesyncd[678]: Timed out waiting for reply from 91.189.91.157:123 (ntp.ubuntu.com).
Nov 17 08:03:41 icluster-node-d000 systemd-timesyncd[678]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).
Nov 17 08:17:01 icluster-node-d000 CRON[2829]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 17 08:43:03 icluster-node-d000 systemd[1]: Created slice User Slice of icluster.
Nov 17 08:43:03 icluster-node-d000 systemd[1]: Starting User Manager for UID 1000...
Nov 17 08:43:03 icluster-node-d000 systemd[1]: Started Session 20 of user icluster.
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Listening on GnuPG network certificate management daemon.
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Reached target Timers.
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Listening on GnuPG cryptographic agent and passphrase cache.
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Reached target Sockets.
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Reached target Paths.
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Reached target Basic System.
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Reached target Default.
Nov 17 08:43:03 icluster-node-d000 systemd[2834]: Startup finished in 33ms.
Nov 17 08:43:03 icluster-node-d000 systemd[1]: Started User Manager for UID 1000.
Nov 17 08:43:24 icluster-node-d000 systemd[1]: Started Elasticsearch.
Nov 17 08:43:24 icluster-node-d000 elasticsearch[2970]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Nov 17 08:43:25 icluster-node-d000 elasticsearch[2970]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 17 08:43:25 icluster-node-d000 elasticsearch[2970]: OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elastic.p12
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elastic.p12
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:145)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at java.base/java.nio.file.Files.walkFileTree(Files.java:2760)
Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: #011at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:218)
[...]

According to this information, elasticsearch starts with a different path:

Nov 17 08:43:27 icluster-node-d000 elasticsearch[2970]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elastic.p12

/etc/elasticsearch/certs/elastic.p12 is old path.

However, the current path in the configuration (yml file) is:

/es_certs/elastic-certificates.p12

Why is elasticsearch not loading the new path?
How to force path update?

2

How about the rest of your configuration ? Do you maybe use /etc/elasticsearch/certs/elastic.p12 in another configuration setting?

Please note that either way, you will need to place the certificate files and/or keystores in a subdirectory of the elasticsearch configuration so that elasticsearch can have read access to them. /es_certs/elastic-certificates.p12 will not work .

In summary:

  • Move your elastic-certificates.p12 in a subdirectory of the elasticsearch configuration.
  • Change your configuration accordingly and share your whole elasticsearch.yml with us
  • Restart the service
  • Share the logs with us
3

Well ...
I only changed the elasticsearch.yml file.

Current file content: elasticsearch.yml:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

#give your cluster a name.
cluster.name: cluster

#give your nodes a name (change node number from node to node).
node.name: cluster-node-d000

#define node 1 as master-eligible:
node.master: true

#define nodes 2 and 3 as data nodes:
node.data: true

#enter the private IP and port of your node:
network.host: 192.168.8.30
http.port: 9200

#detail the private IPs of your nodes:
discovery.zen.ping.unicast.hosts: ["192.168.8.30", "192.168.8.31", "192.168.8.32"]

#xpack.security.enabled: true
#xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
#xpack.security.transport.ssl.keystore.path: /es_certs/elastic-certificates.p12
#xpack.security.transport.ssl.truststore.path: /es_certs/elastic-certificates.p12

I followed this totorial:

Even after restarting the machine and running elasticsearch, it still returns an error.

4

Please :

5

In /var/log/elasticsearch/ elasticsearch create dump only to file gc.log:

[...]
[2019-11-26T15:19:06.830+0000][1896][safepoint    ] Leaving safepoint region
[2019-11-26T15:19:06.897+0000][1896][gc,heap,exit ] Heap
[2019-11-26T15:19:06.897+0000][1896][gc,heap,exit ]  par new generation   total 314560K, used 61547K [0x00000000c0000000, 0x00000000d5550000, 0x00000000d5550000)
[2019-11-26T15:19:06.897+0000][1896][gc,heap,exit ]   eden space 279616K,  22% used [0x00000000c0000000, 0x00000000c3c1acb8, 0x00000000d1110000)
[2019-11-26T15:19:06.897+0000][1896][gc,heap,exit ]   from space 34944K,   0% used [0x00000000d1110000, 0x00000000d1110000, 0x00000000d3330000)
[2019-11-26T15:19:06.897+0000][1896][gc,heap,exit ]   to   space 34944K,   0% used [0x00000000d3330000, 0x00000000d3330000, 0x00000000d5550000)
[2019-11-26T15:19:06.897+0000][1896][gc,heap,exit ]  concurrent mark-sweep generation total 699072K, used 0K [0x00000000d5550000, 0x0000000100000000, 0x0000000100000000)
[2019-11-26T15:19:06.897+0000][1896][gc,heap,exit ]  Metaspace       used 10140K, capacity 10676K, committed 10880K, reserved 1058816K
[2019-11-26T15:19:06.897+0000][1896][gc,heap,exit ]   class space    used 1241K, capacity 1374K, committed 1408K, reserved 1048576K
[2019-11-26T15:19:06.897+0000][1896][safepoint    ] Application time: 0.0671690 seconds
[2019-11-26T15:19:06.897+0000][1896][safepoint    ] Entering safepoint region: Halt

In sysmon logs:

Nov 26 15:19:04 icluster-node-d000 systemd[1]: Started Elasticsearch.
Nov 26 15:19:04 icluster-node-d000 elasticsearch[1896]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Nov 26 15:19:04 icluster-node-d000 elasticsearch[1896]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Nov 26 15:19:04 icluster-node-d000 elasticsearch[1896]: OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elastic.p12
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/certs/elastic.p12
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:145)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/java.nio.file.Files.readAttributes(Files.java:1763)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at java.base/java.nio.file.Files.walkFileTree(Files.java:2760)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:218)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:127)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:302)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.cli.Command.main(Command.java:90)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: #011at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: Refer to the log for complete error details.
Nov 26 15:19:06 icluster-node-d000 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 26 15:19:06 icluster-node-d000 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Nov 26 15:20:46 icluster-node-d000 systemd-timesyncd[697]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Nov 26 15:20:56 icluster-node-d000 systemd-timesyncd[697]: Timed out waiting for reply from 91.189.94.4:123 (ntp.ubuntu.com).
Nov 26 15:20:56 icluster-node-d000 systemd-timesyncd[697]: Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).
Nov 26 15:26:59 icluster-node-d000 systemd-timesyncd[697]: Timed out waiting for reply from 91.189.91.157:123 (ntp.ubuntu.com).
Nov 26 15:27:10 icluster-node-d000 systemd-timesyncd[697]: Timed out waiting for reply from 91.189.89.198:123 (ntp.ubuntu.com).
Nov 26 15:29:18 icluster-node-d000 systemd[1]: Starting Cleanup of Temporary Directories...
Nov 26 15:29:18 icluster-node-d000 systemd[1]: Started Cleanup of Temporary Directories.
Nov 26 15:31:26 icluster-node-d000 systemd-timesyncd[697]: Synchronized to time server 91.189.89.199:123 (ntp.ubuntu.com).

In the logs there is "icluster *" in the file elasticsearch.yml (on the forum), the cluster name is "cluster *" but by pasting the configuration I removed the "and" from the name. It's not a matter of it :wink:

6

... and service status:

icluster@icluster-node-d000:~$ sudo service elasticsearch status
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-11-26 15:19:06 UTC; 38min ago
     Docs: http://www.elastic.co
  Process: 1896 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 1896 (code=exited, status=1/FAILURE)

Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]:         at org.elasticsearch.cli.Command.main(Command.java:90)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Nov 26 15:19:06 icluster-node-d000 elasticsearch[1896]: Refer to the log for complete error details.
Nov 26 15:19:06 icluster-node-d000 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 26 15:19:06 icluster-node-d000 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
7

Are you sure that the elasticsearch.yml you share with us is the one that is located at /etc/elasticsearch/elasticsearch.yml ?

8

Yes:

icluster@icluster-node-d000:~$ sudo cat /etc/elasticsearch/elasticsearch.yml
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

#give your cluster a name.
cluster.name: icluster

#give your nodes a name (change node number from node to node).
node.name: icluster-node-d000

#define node 1 as master-eligible:
node.master: true

#define nodes 2 and 3 as data nodes:
node.data: true

#enter the private IP and port of your node:
network.host: 192.168.8.30
http.port: 9200

#detail the private IPs of your nodes:
discovery.zen.ping.unicast.hosts: ["192.168.8.30", "192.168.8.31", "192.168.8.32"]

#xpack.security.enabled: true
#xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
#xpack.security.transport.ssl.keystore.path: /es_certs/elastic-certificates.p12
#xpack.security.transport.ssl.truststore.path: /es_certs/elastic-certificates.p12
9

How did you install elasticsearch on your server ? Did you use the DEB or RPM package ? if so can you paste the contents of /etc/default/elasticsearch or /etc/sysconfig/elasticsearch and the output of

echo $ES_PATH_CONF

?

10

$ES_PATH_CONF is empty.

icluster@icluster-node-d000:~$ echo $ES_PATH_CONF

icluster@icluster-node-d000:~$ ls -l /etc/sysconfig/elasticsearch
ls: cannot access '/etc/sysconfig/elasticsearch': No such file or directory
icluster@icluster-node-d000:~$ cat /etc/sysconfig/elasticsearch
cat: /etc/sysconfig/elasticsearch: No such file or directory
icluster@icluster-node-d000:~$ ls -l /etc/default/elasticsearch
-rw-rw---- 1 root elasticsearch 1613 Aug 29 19:15 /etc/default/elasticsearch
icluster@icluster-node-d000:~$ cat /etc/default/elasticsearch
cat: /etc/default/elasticsearch: Permission denied
icluster@icluster-node-d000:~$ sudo cat /etc/default/elasticsearch
################################
# Elasticsearch
################################

# Elasticsearch home directory
#ES_HOME=/usr/share/elasticsearch

# Elasticsearch Java path
#JAVA_HOME=

# Elasticsearch configuration directory
ES_PATH_CONF=/etc/elasticsearch

# Elasticsearch PID directory
#PID_DIR=/var/run/elasticsearch

# Additional Java OPTS
#ES_JAVA_OPTS=

# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true

################################
# Elasticsearch service
################################

# SysV init.d
#
# The number of seconds to wait before checking if Elasticsearch started successfully as a daemon process
ES_STARTUP_SLEEP_TIME=5

################################
# System properties
################################

# Specifies the maximum file descriptor number that can be opened by this process
# When using Systemd, this setting is ignored and the LimitNOFILE defined in
# /usr/lib/systemd/system/elasticsearch.service takes precedence
#MAX_OPEN_FILES=65535

# The maximum number of bytes of memory that may be locked into RAM
# Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option
# in elasticsearch.yml.
# When using systemd, LimitMEMLOCK must be set in a unit file such as
# /etc/systemd/system/elasticsearch.service.d/override.conf.
#MAX_LOCKED_MEMORY=unlimited

# Maximum number of VMA (Virtual Memory Areas) a process can own
# When using Systemd, this setting is ignored and the 'vm.max_map_count'
# property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
#MAX_MAP_COUNT=262144
11

Please answer all relevant questions:

12

Oh sorry.

I install via apt-get install and repo:

deb https://artifacts.elastic.co/packages/6.x/apt stable main

Everything was working well before the changes in yml file (security records).

13

Hi @Rysiu

Please share the permissions on this file (/etc/elasticsearch/certs/elastic.p12 with us. Elasticsearch does not have access to this file.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.