Elasticsearch 6.x - CentOS7 - unable to apply custom path.data and path.logs

Hi All,

RPM (yum) Installed Elasticsearch 6.8.0 (single instance) on CentOS 7.6 Server.

The installation with default configuration tested successfully using systemctl command.

GOAL: Save path.data and path.logs in 2 locations:

  1. Default Locations: /var/lib/elasticsearch & /var/log/elasticsearch
  2. Directories in another user profile (user_name): /home/user_name/exxxxxxxxxxxx680/data & /home/user_name/exxxxxxxxxxxx680/logs

.yml file updated as follows with custom path.logs and path.data with the various syntax options available; all have been tried; however, none work and Elasticsearch service (systemctl) exits with FAILED status:

> # ---------------------------------- Cluster -----------------------------------
> cluster.name: px-xx-cxxxxxxx
> #
> # ------------------------------------ Node ------------------------------------
> node.name: px-xx-nxxxx
> #
> # Add custom attributes to the node:
> #
> #node.attr.rack: r1
> #
> # ----------------------------------- Paths ------------------------------------
> #
> # Path to directory where to store the data (separate multiple locations by comma):
> #
> # Path to data files:
> #
> #path.data: /var/lib/elasticsearch
> #path.data: /home/user_name/exxxxxxxxxxxx680/data
> #path.data: /var/lib/elasticsearch,/home/user_name/exxxxxxxxxxxx680/data
> #path.data: ["/var/lib/elasticsearch", "/home/user_name/exxxxxxxxxxxx680/data"]
> #path.data: /home/user_name/exxxxxxxxxxxx680/data
> #path.data: ['/var/lib/elasticsearch', '/home/user_name/exxxxxxxxxxxx680/data']
> path.data: /home/user_name/exxxxxxxxxxxx680/data
> #
> # Path to log files:
> #
> #path.logs: /var/log/elasticsearch
> #path.logs:/home/user_name/exxxxxxxxxxxx680/logs
> #path.logs: /var/log/elasticsearch,/home/user_name/exxxxxxxxxxxx680/logs
> #path.logs: ["/var/log/elasticsearch", "/home/user_name/exxxxxxxxxxxx680/logs"]
> #path.logs:/home/user_name/exxxxxxxxxxxx680/logs
> #path.logs: ['/var/log/elasticsearch', '/home/user_name/exxxxxxxxxxxx680/logs']
> path.logs: /home/user_name/exxxxxxxxxxxx680/logs
> #
> #
> #path:
> #  data:
> #    - /var/lib/elasticsearch
> #    - /home/user_name/exxxxxxxxxxxx680/data
> #  logs:
> #    - /var/log/elasticsearch
> #    - /home/user_name/exxxxxxxxxxxx680/logs
>
> # ----------------------------------- Memory -----------------------------------
> #
> # Lock the memory on startup:
> #
> #bootstrap.memory_lock: true
> #
> # Make sure that the heap size is set to about half the memory available
> # on the system and that the owner of the process is allowed to use this
> # limit.
> #
> # Elasticsearch performs poorly when the system is swapping the memory.
> #
> # ---------------------------------- Network -----------------------------------
> #
> # Set the bind address to a specific IP (IPv4 or IPv6):
> #
> #network.host: 192.168.0.1
> network.host: 1xx.1xx.xxx.xxx
> #
> # Set a custom port for HTTP:
> #
> #http.port: xxx0
> http.port: xxx0
> #
> # For more information, consult the network module documentation.
> #
> # --------------------------------- Discovery ----------------------------------
> #
> # Pass an initial list of hosts to perform discovery when new node is started:
> # The default list of hosts is ["127.0.0.1", "[::1]"]
> #
> #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
> #
> # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
> #
> #discovery.zen.minimum_master_nodes: 
> #
> # For more information, consult the zen discovery module documentation.
> #
> # ---------------------------------- Gateway -----------------------------------
> #
> # Block initial recovery after a full cluster restart until N nodes are started:
> #
> #gateway.recover_after_nodes: 3
> #
> # For more information, consult the gateway module documentation.
> #
> # ---------------------------------- Various -----------------------------------
> #
> # Require explicit names when deleting indices:
> #
> #action.destructive_requires_name: true

No Elasticsearch logs are generated when execution done with above configuration.

systemctl status elasticsearch:

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2019-07-16 11:14:53 IST; 5s ago
     Docs: http://www.elastic.co
  Process: 31202 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid (code=exited, status=1/FAILURE)
 Main PID: 31202 (code=exited, status=1/FAILURE)

Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:...
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:...
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:...
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:...
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:...
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:...
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:...
Jul 16 11:14:53 localhost.localdomain systemd[1]: elasticsearch.service: main...
Jul 16 11:14:53 localhost.localdomain systemd[1]: Unit elasticsearch.service ...
Jul 16 11:14:53 localhost.localdomain systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

The journalctl logs are in the following post due to post size limitations.

Custom Destination Directory CHMOD set to 777.

I have tried all possible combinations of permissions settings on the custom destination directory; i.e.:

root:root
root:elasticsearch
elasticsearch:elasticsearch
user:elasticsearch
user:root
user:user

...but no joy; currently set as follows

drwxrwxrwx. 4 root elasticsearch 42 Jul 15 16:51 exxxxxxxxxxxxxxx

We need to get this up and running in our production environment asap; Please Help!

Thanks in advance!
Regards,
JK

journalctl extract:

 -- Logs begin at Fri 2019-07-05 19:31:57 IST, end at Tue 2019-07-16 11:16:53 IST. --
    Jul 16 11:14:50 localhost.localdomain systemd[1]: Started Elasticsearch.
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:53,380 main ERROR Unable to create file /home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx.log java.io.IOException: Could not create directory /home/user_name/exxxxxxxxxxxx680/logs
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.util.FileUtils.mkdir(FileUtils.java:127)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:144)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:627)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
     .....org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:53,385 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@52b1beb6] unable to create manager for [/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@273e7444[pattern=/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %.-10000m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@52b1beb6] unable to create manager for [/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@273e7444[pattern=/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx-%d{yyyy-MM-dd}-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true), SizeBasedTriggeringPolicy(size=134217728)]), strategy=DefaultRolloverStrategy(min=-2147483648, max=2147483647, useMax=false), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %.-10000m%n, filePermissions=null, fileOwner=null]]
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
       .....org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:53,386 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
    Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
      .....org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:53,398 main ERROR Unable to create file /home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx_audit.log java.io.IOException: Could not create directory /home/user_name/exxxxxxxxxxxx680/logs
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.util.FileUtils.mkdir(FileUtils.java:127)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:144)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:627)
   .....org.elasticsearch.cli.Command.main(Command.java:90)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:53,400 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@52b1beb6] unable to create manager for [/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx_audit.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@229c6181[pattern=/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx_audit-%d{yyyy-MM-dd}.log, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[TimeBasedTriggeringPolicy(nextRolloverMillis=0, interval=1, modulate=true)]), strategy=DefaultRolloverStrategy(min=1, max=7, useMax=true), advertiseURI=null, layout={"@timestamp":"%d{ISO8601}"%varsNotEmpty{, "node.name":"%enc{%map{node.name}}{JSON}"}%varsNotEmpty{, "node.id":"%enc{%map{node.id}}{JSON}"}%varsNotEmpty{, "host.name":"%enc{%map{host.name}}{JSON}"}%varsNotEmpty{, "host.ip":"%enc{%map{host.ip}}{JSON}"}%varsNotEmpty{, "event.type":"%enc{%map{event.type}}{JSON}"}%varsNotEmpty{, "event.action":"%enc{%map{event.action}}{JSON}"}%varsNotEmpty{, "user.name":"%enc{%map{user.name}}{JSON}"}%varsNotEmpty{, 
....
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
   .....org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:53,401 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
  .....)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:53,404 main ERROR Unable to create file /home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx_deprecation.log java.io.IOException: Could not create directory /home/user_name/exxxxxxxxxxxx680/logs
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.util.FileUtils.mkdir(FileUtils.java:127)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:144)
  ......
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Jul 16 11:14:53 localhost.localdomain elasticsearch[31202]: 2019-07-16 11:14:53,405 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@52b1beb6] unable to create manager for [/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@38cee291[pattern=/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %.-10000m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@52b1beb6] unable to create manager for [/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@38cee291[pattern=/home/user_name/exxxxxxxxxxxx680/logs/px-xx-cxxxxxxx_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %.-10000m%n, filePermissions=null, fileOwner=null]]

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.