Elastic service wont start

alvaroca1 · August 9, 2016, 9:24pm

Good day,
I'm new to Linux and New to elastic I'm attempting an installation on Red Hat server. elastic installed and I was able to create the service however when I try to start the service it stops immediately. at fist I believed it to be permission on the location of the data files and log file so I gave permissions to my elastic user to write to the new location it still fails.

Started Elasticsearch.
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: log4j:ERROR setFile(null,true) call failed.
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: java.io.FileNotFoundException: /eslogs/elasticlogs/es-winlogs.log (Permission denied)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at java.io.FileOutputStream.open0(Native Method)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at java.io.FileOutputStream.(FileOutputStream.java:213)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at java.io.FileOutputStream.(FileOutputStream.java:133)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:128)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:243)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: log4j:ERROR Either File or DatePattern options are not set for appender [file].
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: log4j:ERROR setFile(null,true) call failed.

alvaroca1 · August 9, 2016, 9:25pm

service elasticsearch start
Starting elasticsearch (via systemctl): ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: elasticadmin
Password:
==== AUTHENTICATION COMPLETE ===
[ OK ]
[elasticadmin@ESSEP01 ~]$ service --status-all | grep elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Process: 2923 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 2921 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: [2016-08-09 16:51:25,668][INFO ][plugins ] [essep01] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: Exception in thread "main" java.lang.IllegalStateException: Failed to created node environment
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: Likely root cause: java.nio.file.AccessDeniedException: /apps/elasticdata/es-winlogs/nodes
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
Aug 09 16:51:25 ESSEP01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Aug 09 16:51:25 ESSEP01 systemd[1]: Unit elasticsearch.service entered failed state.
Aug 09 16:51:25 ESSEP01 systemd[1]: elasticsearch.service failed.

warkolm · August 9, 2016, 9:25pm

It still looks like a permissions error.

How did you install ES? What permissions are on the directory?

warkolm · August 9, 2016, 9:29pm

Please format that with the </> button, it's impossible to read.

alvaroca1 · August 9, 2016, 9:29pm

[elasticadmin@ESSEP01 es-winlogs]$ ls -ltra
total 8
drwxrwxr-x. 3 elasticadmin elasticadmin 4096 Aug 9 16:22 ..
-rw-r--r--. 1 elasticadmin elasticadmin 0 Aug 9 17:00 nodes
drwxrwxr-x. 2 elasticadmin elasticadmin 4096 Aug 9 17:00 .

[elasticadmin@ESSEP01 eslogs]$ ls -ltra
total 28
drwx------. 2 root root 16384 Aug 9 11:20 lost+found
dr-xr-xr-x. 19 root root 4096 Aug 9 11:20 ..
drwxrwxr-x. 4 elasticadmin elasticadmin 4096 Aug 9 15:57 .
drwxrwxr-x. 2 elasticadmin elasticadmin 4096 Aug 9 17:01 elasticlogs

alvaroca1 · August 9, 2016, 9:34pm

this is the YML file with only the items modified 

# Use a descriptive name for your cluster:
#
 cluster.name: "es-winlogs"
# Use a descriptive name for the node:
#
 node.name: essep01

# Path to directory where to store the data (separate multiple locations by comma):
#
 path.data: /apps/elasticdata
#
# Path to log files:
#
 path.logs: /eslogs/elasticlogs

# Set the bind address to a specific IP (IPv4 or IPv6):
#
 network.host: localhost

warkolm · August 9, 2016, 10:19pm

And what user are you running ES as? How are you starting it?

alvaroca1 · August 9, 2016, 11:22pm

[elasticadmin@ESSEP01 ~]$ service elasticsearch start
Starting elasticsearch (via systemctl): ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: elasticadmin
Password:
==== AUTHENTICATION COMPLETE ===

warkolm · August 10, 2016, 1:12am

If you installed ES as a DEB/RPM, then it'll use the elasticsearch user.

alvaroca1 · August 10, 2016, 12:15pm

I'm new to Linux and that is what I am trying to use the user I created "elasticadmin" to start the service but unless I use sudo the service says it does not have permission on its own.... can you explain a bit more.

magnusbaeck · August 10, 2016, 12:42pm

Which user runs service ... start doesn't matter. It's the Elasticsearch configuration that determines what user the service runs as, and unless configured otherwise Elasticsearch will start as the elasticsearch user. Until you know better, stick to the defaults. You will most likely not need an elasticadmin user.

alvaroca1 · August 10, 2016, 5:17pm

magnus can you elaborate further... I did the install based on the instructions on the elastic site I haven't deviated from that. and my install is plane

magnusbaeck · August 10, 2016, 5:32pm

Neither /apps/elasticdata nor /eslogs are standard directories so I'm pretty sure you've deviated somehow. Anyway, make sure the directories that ES uses are owned by the elasticsearch user and nothing else.

alvaroca1 · August 10, 2016, 5:58pm

correct I moved the paths so that we can have separate disk for them, is that a bad thing?

also I saw this on the logs

Aug 10 13:54:38 ESSEP01 elasticsearch[4996]: log4j:ERROR Either File or DatePattern options are not set for appender [index_search_slow_log_file].
Aug 10 13:54:38 ESSEP01 elasticsearch[4996]: [2016-08-10 13:54:38,760][INFO ][node ] [essep01] version[2.3.5], pid[4996], build[90f439f/2016-07-27T10:36:52Z]
Aug 10 13:54:38 ESSEP01 elasticsearch[4996]: [2016-08-10 13:54:38,761][INFO ][node ] [essep01] initializing ...
Aug 10 13:54:39 ESSEP01 elasticsearch[4996]: [2016-08-10 13:54:39,179][INFO ][plugins ] [essep01] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
Aug 10 13:54:39 ESSEP01 elasticsearch[4996]: Exception in thread "main" java.lang.IllegalStateException: Failed to created node environment
Aug 10 13:54:39 ESSEP01 elasticsearch[4996]: Likely root cause: java.nio.file.FileSystemException: /apps/elasticdata/es-winlogs/nodes/0: Not a directory

magnusbaeck · August 10, 2016, 6:03pm

correct I moved the paths so that we can have separate disk for them, is that a bad thing?

Not at all.

Aug 10 13:54:39 ESSEP01 elasticsearch[4996]: Likely root cause: java.nio.file.FileSystemException: /apps/elasticdata/es-winlogs/nodes/0: Not a directory

I wonder if that directory has ended up in a weird state. Perhaps wipe it to start fresh?

alvaroca1 · August 10, 2016, 7:45pm

so it seems the local user created by the rpm installer "elasticsearch" was the culprit. and I can now curl the local install using local host, however I need to allow external connection and I don't have IP tables.

[elasticadmin@ESSEP01 elasticdata]$ curl -XGET "http://192.168.10.80:9200/_cluster/state"
curl: (7) Failed connect to 192.168.10.80:9200; Connection refused

[elasticadmin@ESSEP01 elasticdata]$ curl -XGET "http://localhost:9200"
{
"name" : "essep01",
"cluster_name" : "es-winlogs",
"version" : {
"number" : "2.3.5",
"build_hash" : "90f439ff60a3c0f497f91663701e64ccd01edbb4",
"build_timestamp" : "2016-07-27T10:36:52Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
[elasticadmin@ESSEP01 elasticdata]$ curl -XGET "http://localhost:9200/_cluster/state"
{"cluster_name":"es-winlogs","version":2,"state_uuid":"jlgiMfqMSuCaB7--Qr9UuA","master_node":"eX3oouQVlhost/127.0.0.1:9300","attributes":{}}},"metadata":{"cluster_uuid":"HLXurY_1T5u54plc8jw1tQ","templatesindent preformatted text by 4 spaces

magnusbaeck · August 11, 2016, 3:32am

Have a look at the network.host option for elasticsearch.yml.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

alvaroca1 · August 11, 2016, 6:23pm

so I made a change as recommended on the config file and now it wont start.

Aug 11 13:40:10 ESSEP01 systemd[1]: Started Elasticsearch.
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: Exception in thread "main" SettingsException[Failed to load settings from [
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: in 'reader', line 54, column 2:
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: network.host:[inet, _local]
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: ^
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: could not find expected ':'
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: in 'reader', line 55, column 1:
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: #
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: network.host:[inet, _local]
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: ^
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: could not find expected ':'
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: in 'reader', line 55, column 1:
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: #
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: ^
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: ];
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: Likely root cause: while scanning a simple key
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: in 'reader', line 54, column 2:
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: network.host:[inet, _local]

magnusbaeck · August 11, 2016, 6:29pm

What if you put a space after the colon?

alvaroca1 · August 22, 2016, 4:26pm

this is resolved thanks, it was blank spaces on the config

Topic		Replies	Views
Elasticsearch unable to start Elasticsearch	11	26213	November 4, 2022
Elasticsearch fails to start Elasticsearch	4	1154	January 9, 2019
Elasticsearch wont start on Ubuntu 14.04 Elasticsearch	10	1004	July 6, 2017
Elasticsearch service doesn't start upon system reboot Elasticsearch	4	4876	September 13, 2017
Elasticsearch service start Elasticsearch	17	7327	July 5, 2017

Elastic service wont start

Related topics