Elastic service wont start

Elastic Stack Elasticsearch
1

Good day,
I'm new to Linux and New to elastic I'm attempting an installation on Red Hat server. elastic installed and I was able to create the service however when I try to start the service it stops immediately. at fist I believed it to be permission on the location of the data files and log file so I gave permissions to my elastic user to write to the new location it still fails.

Started Elasticsearch.
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: log4j:ERROR setFile(null,true) call failed.
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: java.io.FileNotFoundException: /eslogs/elasticlogs/es-winlogs.log (Permission denied)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at java.io.FileOutputStream.open0(Native Method)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at java.io.FileOutputStream.open(FileOutputStream.java:270)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at java.io.FileOutputStream.(FileOutputStream.java:213)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at java.io.FileOutputStream.(FileOutputStream.java:133)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:128)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:243)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: log4j:ERROR Either File or DatePattern options are not set for appender [file].
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: log4j:ERROR setFile(null,true) call failed.

2

service elasticsearch start
Starting elasticsearch (via systemctl): ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: elasticadmin
Password:
==== AUTHENTICATION COMPLETE ===
[ OK ]
[elasticadmin@ESSEP01 ~]$ service --status-all | grep elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Process: 2923 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 2921 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: [2016-08-09 16:51:25,668][INFO ][plugins ] [essep01] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: Exception in thread "main" java.lang.IllegalStateException: Failed to created node environment
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: Likely root cause: java.nio.file.AccessDeniedException: /apps/elasticdata/es-winlogs/nodes
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
Aug 09 16:51:25 ESSEP01 elasticsearch[2923]: at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
Aug 09 16:51:25 ESSEP01 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Aug 09 16:51:25 ESSEP01 systemd[1]: Unit elasticsearch.service entered failed state.
Aug 09 16:51:25 ESSEP01 systemd[1]: elasticsearch.service failed.

3

It still looks like a permissions error.

How did you install ES? What permissions are on the directory?

5

Please format that with the </> button, it's impossible to read.

6

[elasticadmin@ESSEP01 es-winlogs]$ ls -ltra
total 8
drwxrwxr-x. 3 elasticadmin elasticadmin 4096 Aug 9 16:22 ..
-rw-r--r--. 1 elasticadmin elasticadmin 0 Aug 9 17:00 nodes
drwxrwxr-x. 2 elasticadmin elasticadmin 4096 Aug 9 17:00 .

[elasticadmin@ESSEP01 eslogs]$ ls -ltra
total 28
drwx------. 2 root root 16384 Aug 9 11:20 lost+found
dr-xr-xr-x. 19 root root 4096 Aug 9 11:20 ..
drwxrwxr-x. 4 elasticadmin elasticadmin 4096 Aug 9 15:57 .
drwxrwxr-x. 2 elasticadmin elasticadmin 4096 Aug 9 17:01 elasticlogs

9 
this is the YML file with only the items modified 

# Use a descriptive name for your cluster:
#
 cluster.name: "es-winlogs"
# Use a descriptive name for the node:
#
 node.name: essep01

# Path to directory where to store the data (separate multiple locations by comma):
#
 path.data: /apps/elasticdata
#
# Path to log files:
#
 path.logs: /eslogs/elasticlogs

# Set the bind address to a specific IP (IPv4 or IPv6):
#
 network.host: localhost
10

And what user are you running ES as? How are you starting it?

11

[elasticadmin@ESSEP01 ~]$ service elasticsearch start
Starting elasticsearch (via systemctl): ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: elasticadmin
Password:
==== AUTHENTICATION COMPLETE ===

12

If you installed ES as a DEB/RPM, then it'll use the elasticsearch user.

13

I'm new to Linux and that is what I am trying to use the user I created "elasticadmin" to start the service but unless I use sudo the service says it does not have permission on its own.... can you explain a bit more.

14

Which user runs service ... start doesn't matter. It's the Elasticsearch configuration that determines what user the service runs as, and unless configured otherwise Elasticsearch will start as the elasticsearch user. Until you know better, stick to the defaults. You will most likely not need an elasticadmin user.

15

magnus can you elaborate further... I did the install based on the instructions on the elastic site I haven't deviated from that. and my install is plane

16

Neither /apps/elasticdata nor /eslogs are standard directories so I'm pretty sure you've deviated somehow. Anyway, make sure the directories that ES uses are owned by the elasticsearch user and nothing else.

17

correct I moved the paths so that we can have separate disk for them, is that a bad thing?

also I saw this on the logs

Aug 10 13:54:38 ESSEP01 elasticsearch[4996]: log4j:ERROR Either File or DatePattern options are not set for appender [index_search_slow_log_file].
Aug 10 13:54:38 ESSEP01 elasticsearch[4996]: [2016-08-10 13:54:38,760][INFO ][node ] [essep01] version[2.3.5], pid[4996], build[90f439f/2016-07-27T10:36:52Z]
Aug 10 13:54:38 ESSEP01 elasticsearch[4996]: [2016-08-10 13:54:38,761][INFO ][node ] [essep01] initializing ...
Aug 10 13:54:39 ESSEP01 elasticsearch[4996]: [2016-08-10 13:54:39,179][INFO ][plugins ] [essep01] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
Aug 10 13:54:39 ESSEP01 elasticsearch[4996]: Exception in thread "main" java.lang.IllegalStateException: Failed to created node environment
Aug 10 13:54:39 ESSEP01 elasticsearch[4996]: Likely root cause: java.nio.file.FileSystemException: /apps/elasticdata/es-winlogs/nodes/0: Not a directory

18

correct I moved the paths so that we can have separate disk for them, is that a bad thing?

Not at all.

Aug 10 13:54:39 ESSEP01 elasticsearch[4996]: Likely root cause: java.nio.file.FileSystemException: /apps/elasticdata/es-winlogs/nodes/0: Not a directory

I wonder if that directory has ended up in a weird state. Perhaps wipe it to start fresh?

19

so it seems the local user created by the rpm installer "elasticsearch" was the culprit. and I can now curl the local install using local host, however I need to allow external connection and I don't have IP tables.

[elasticadmin@ESSEP01 elasticdata]$ curl -XGET "http://192.168.10.80:9200/_cluster/state"
curl: (7) Failed connect to 192.168.10.80:9200; Connection refused

[elasticadmin@ESSEP01 elasticdata]$ curl -XGET "http://localhost:9200"
{
"name" : "essep01",
"cluster_name" : "es-winlogs",
"version" : {
"number" : "2.3.5",
"build_hash" : "90f439ff60a3c0f497f91663701e64ccd01edbb4",
"build_timestamp" : "2016-07-27T10:36:52Z",
"build_snapshot" : false,
"lucene_version" : "5.5.0"
},
"tagline" : "You Know, for Search"
}
[elasticadmin@ESSEP01 elasticdata]$ curl -XGET "http://localhost:9200/_cluster/state"
{"cluster_name":"es-winlogs","version":2,"state_uuid":"jlgiMfqMSuCaB7--Qr9UuA","master_node":"eX3oouQVlhost/127.0.0.1:9300","attributes":{}}},"metadata":{"cluster_uuid":"HLXurY_1T5u54plc8jw1tQ","templatesindent preformatted text by 4 spaces

20

Have a look at the network.host option for elasticsearch.yml.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

21

so I made a change as recommended on the config file and now it wont start.

Aug 11 13:40:10 ESSEP01 systemd[1]: Started Elasticsearch.
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: Exception in thread "main" SettingsException[Failed to load settings from [
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: in 'reader', line 54, column 2:
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: network.host:[inet, _local]
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: ^
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: could not find expected ':'
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: in 'reader', line 55, column 1:
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: #
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: network.host:[inet, _local]
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: ^
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: could not find expected ':'
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: in 'reader', line 55, column 1:
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: #
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: ^
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: ];
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: Likely root cause: while scanning a simple key
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: in 'reader', line 54, column 2:
Aug 11 13:40:12 ESSEP01 elasticsearch[1010]: network.host:[inet, _local]

22

What if you put a space after the colon?

23

this is resolved thanks, it was blank spaces on the config