Elasticsearch cannot start when change the default path of log and data


(sahere rahimi) #1

hi all,
I install elasticsearch using RPM version. first i used default path of log and data as following ,

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

when i used default path, elasticsearch can start successfully by following command:
systemctl start elasticsearch.service

but when I change the path of log and data as following, elasticseach cannot start and no logs be created in the defined path
path.data: /home/srahimi/logelk
path.logs: /home/srahimi/logelk

could you please advise me?


(Harsh Bajaj) #2

Hi @sahere37,

Please check the permissions of that folder. if all ok then please share the logs of elasticsearch. so that i can identify the problem.

Regards,
Harsh Bajaj


(sahere rahimi) #3

the defined folder needs no permission. its is the home directory.
it is noted that the path (/var/log/elasticsearch) needs sudo permission but the defined path has no one. maybe, the path which we define for storing logs and data in RPM mode should have permission??


(Harsh Bajaj) #4

Could you please share the elasticsearch logs so that i can see the actual problem.


(sahere rahimi) #5

thanks for replying.
as i said, when i use default path (var/log/elasticsearch), everything is ok and elasticsearch.log is as following:

[2018-11-28T02:27:50,488][INFO ][o.e.n.Node ] [node-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.GLvLpTRo, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [analysis-common]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-common]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-expression]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-mustache]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-painless]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [mapper-extras]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [parent-join]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [percolator]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [rank-eval]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [reindex]
[2018-11-28T02:27:53,431][INFO ][o.e.p.PluginsService ] [node-1] loaded module [repository-url]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty4]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [tribe]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ccr]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-core]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-deprecation]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-graph]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-logstash]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-ml]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-monitoring]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-rollup]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-security]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-sql]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-upgrade]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] loaded module [x-pack-watcher]
[2018-11-28T02:27:53,432][INFO ][o.e.p.PluginsService ] [node-1] no plugins loaded
[2018-11-28T02:28:00,145][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2018-11-28T02:28:00,952][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [node-1] [controller/3996] [Main.cc@109] controller (64 bit): Version 6.5.1 (Build 1c5fe241dd9aea) Copyright (c) 2018 Elasticsearch BV
[2018-11-28T02:28:01,548][DEBUG][o.e.a.ActionModule ] [node-1] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-11-28T02:28:01,860][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen] and host providers [settings]
[2018-11-28T02:28:03,111][INFO ][o.e.n.Node ] [node-1] initialized
[2018-11-28T02:28:03,111][INFO ][o.e.n.Node ] [node-1] starting ...
[2018-11-28T02:28:03,333][INFO ][o.e.t.TransportService ] [node-1] publish_address {192.168.170.137:9300}, bound_addresses {192.168.170.137:9300}
[2018-11-28T02:28:03,370][INFO ][o.e.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-11-28T02:28:06,540][INFO ][o.e.c.s.MasterService ] [node-1] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {node-1}{0nnC4wUtRjmaA40wl-LyhQ}{_w9pZt5KSCWWiAf7mQDHrA}{192.168.170.137}{192.168.170.137:9300}{ml.machine_memory=3873095680, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-11-28T02:28:06,545][INFO ][o.e.c.s.ClusterApplierService] [node-1] new_master {node-1}{0nnC4wUtRjmaA40wl-LyhQ}{_w9pZt5KSCWWiAf7mQDHrA}{192.168.170.137}{192.168.170.137:9300}{ml.machine_memory=3873095680, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {node-1}{0nnC4wUtRjmaA40wl-LyhQ}{_w9pZt5KSCWWiAf7mQDHrA}{192.168.170.137}{192.168.170.137:9300}{ml.machine_memory=3873095680, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-11-28T02:28:06,669][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] publish_address {192.168.170.137:9200}, bound_addresses {192.168.170.137:9200}
[2018-11-28T02:28:06,669][INFO ][o.e.n.Node ] [node-1] started
[2018-11-28T02:28:07,024][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [node-1] Failed to clear cache for realms []
[2018-11-28T02:28:07,083][INFO ][o.e.l.LicenseService ] [node-1] license [2b3ea0e4-fbaa-42e0-961e-a4ec36ac1793] mode [basic] - valid
[2018-11-28T02:28:07,103][INFO ][o.e.g.GatewayService ] [node-1] recovered [0] indices into cluster_state
[2018-11-28T02:29:19,595][INFO ][o.e.n.Node ] [node-1] stopping ...
[2018-11-28T02:29:19,680][INFO ][o.e.x.w.WatcherService ] [node-1] stopping watch service, reason [shutdown initiated]
[2018-11-28T02:29:20,053][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [node-1] [controller/3996] [Main.cc@148] Ml controller exiting
[2018-11-28T02:29:20,057][INFO ][o.e.x.m.j.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started


(sahere rahimi) #6

but, when i use my defined path (/home/srahimi/logelk) elasticsearch cannot start, and more importantly, there is no log in (/home/srahimi/logelk)


(Harsh Bajaj) #7

could you please check "var/log/messages" and share while starting elasticsearch with defined path.


(sahere rahimi) #8

Nov 28 03:22:11 localhost elasticsearch: 2018-11-28 03:22:11,387 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@5c671d7f] unable to create manager for [/home/srahimi/log/elasticsearch_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@313b2ea6[pattern=/home/srahimi/log/elasticsearch_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %.-10000m%n, filePermissions=null, fileOwner=null]] java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory@5c671d7f] unable to create manager for [/home/srahimi/log/elasticsearch_deprecation.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingFileManager$FactoryData@313b2ea6[pattern=/home/srahimi/log/elasticsearch_deprecation-%i.log.gz, append=true, bufferedIO=true, bufferSize=8192, policy=CompositeTriggeringPolicy(policies=[SizeBasedTriggeringPolicy(size=1073741824)]), strategy=DefaultRolloverStrategy(min=1, max=4, useMax=true), advertiseURI=null, layout=[%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %.-10000m%n, filePermissions=null, fileOwner=null]]
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:234)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:127)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:302)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.cli.Command.main(Command.java:90)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
Nov 28 03:22:11 localhost elasticsearch: 2018-11-28 03:22:11,388 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:235)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)


(sahere rahimi) #9

Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:234)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:127)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:302)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.cli.Command.main(Command.java:90)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
Nov 28 03:22:11 localhost elasticsearch: 2018-11-28 03:22:11,389 main ERROR Unable to create file /home/srahimi/log/elasticsearch_index_search_slowlog.log java.io.IOException: Could not create directory /home/srahimi/log
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.util.FileUtils.mkdir(FileUtils.java:127)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:144)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:627)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:608)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:188)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:145)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:61)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:123)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:959)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:899)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:891)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:514)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:238)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547)
Nov 28 03:22:11 localhost elasticsearch: at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:234)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:127)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:302)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.cli.Command.main(Command.java:90)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Nov 28 03:22:11 localhost elasticsearch: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
Nov 28 03:22:11 localhost elasticsearch: 2018-11-28 03:22:11,390 main ERROR Could not create plugin of type class


(Harsh Bajaj) #10

Hi,
According to your logs, im able to see that there is some permission problem to create the defined file.

Request you to provide the proper permissions and check,

/home/srahimi/log/elasticsearch_index_search_slowlog.log java.io.IOException: Could not create directory /home/srahimi/log

Regards,
Harsh