X-pack questions on installation

(Jason Johnson) #1

Greetings - I have just installed x-pack 6.2.4 with ELK versions 6.2.4. I created the user passwords for elastic, kibana, and logstash-system. Kibana is running and I'm able to login - however I am not receiving any new index data. I've confirmed that my default index is * and have even refreshed the fields. One guess I have is that i had to create the elastic user before I installed x-pack - meaning that the elastic users was NOT used to create and install the entire ELK stack. Are there permissions issues here? The elastic user is a super user so i would guess not? I am running on the supplied trial license which should enable all functionality correct?

(Jason Johnson) #2

so this may be telling:

root@oc-elk:~# curl -k -u elastic 'http://<your_server_here>:9200/_xpack/security/_authenticate?pretty'
Enter host password for user 'elastic':
curl: (6) Could not resolve host: <your_server_here>

(Ioannis Kakavas) #3



You are supposed to replace <your_server_here> with the IP Address or the FQDN of the host where Elasticsearch runs.

however I am not receiving any new index data

What exactly do you mean by this ? How are you trying to ingest data into Elasticsearch ? You can try and follow this example in order to load some data into an index and visualize it in Kibana

(Jason Johnson) #4

thanks for the quick reply - duh on the server IP:

root@oc-elk:/etc/kibana# curl -k -u elastic '' Enter host password for user 'elastic':
"username" : "elastic",
"roles" : [
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
"enabled" : true

I am using logstash + filebeat, winlogbeat, metricbeat

(Jason Johnson) #5

so im guessing according to this: https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html

I'll need to add usernames/passwords to each product .yml file to get started?

(Ioannis Kakavas) #6

Yes, each product that connects to Elasticsearch needs to have x-pack installed and you need to configure the necessary credentials so that it can authenticate to Elasticsearch.

(Jason Johnson) #7

Ioannis thank you. I'm configuring each product now - does elasticsearch need a username and password itself configured in the elasticsearch.yml?

(Ioannis Kakavas) #8

No, it doesn't. Go through our documentation in the link you posted, it should get you up and running in no time. Then also go through this section that will talk about setting up TLS for the http and transport layer so that you can have integrity and confidentiality in the communications between the components of the elastic stack

(Jason Johnson) #9

Thanks again - this is a single-node install ....for now. I would guess the TLS portion would not apply. I will review the docs again and follow up later!

(Jason Johnson) #10

I believe I have included the correct usernames and passwords however it seems that im still not getting index data to load.

Here are the 3 configs without commented lines:

root@oc-elk:/etc/elasticsearch# cat elasticsearch.yml | grep -v "#"
cluster.name: ocs-elk-cluster
node.name: oc-elk
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

root@oc-elk:/etc/logstash# cat logstash.yml | grep -v "#"
path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: xxxxxx

root@oc-elk:/etc/kibana# cat kibana.yml | grep -v "#"
server.host: ""
server.name: "oc-elk"
elasticsearch.url: ""
elasticsearch.username: "kibana"
elasticsearch.password: "xxxxx"

I think it's interesting that the logstash info does not have quotes but that is per the documentation.

(Jason Johnson) #11

also interesting that upon remove of x-pack it seems the "plugin" is not installed in logstash:

root@oc-elk:~# service elasticsearch stop
root@oc-elk:~# cd /usr/share/elasticsearch/
root@oc-elk:/usr/share/elasticsearch# bin/elasticsearch-plugin remove x-pack
-> removing [x-pack]...
-> preserving plugin config files [/etc/elasticsearch/x-pack] in case of upgrade; use --purge if not needed
root@oc-elk:/usr/share/elasticsearch# service elasticsearch restart
root@oc-elk:/usr/share/elasticsearch# cd ../kibana/
root@oc-elk:/usr/share/kibana# bin/kibana-plugin remove x-pack
Removing x-pack...
root@oc-elk:/usr/share/kibana# service kibana restart
root@oc-elk:/usr/share/kibana# cd ../logstash/
root@oc-elk:/usr/share/logstash# bin/logstash-plugin remove x-pack
ERROR: Operation aborted, cannot remove plugin., message: This plugin has not been previously installed
root@oc-elk:/usr/share/logstash# service logstash restart

(Ioannis Kakavas) #12

Quotes in YAML are optional when the string value doesn't contain any special characters.

It looks like you haven't installed x-pack for logstash ( You need to install X-Pack for Elasticsearch, Kibana and Logstash ). Please reinstall X-Pack in elasticsearch and then also run

bin/logstash-plugin install x-pack

(Jason Johnson) #13

i think there are a few things wrong here - first my elasticsearch and java was installed as root. I've been unable to get x-pack running at all. I've removed x-pack entirely and will try again.

I've added the user elastic to sudo and am trying the x-pack install now.

installing the machine learning engine is resulting in this:

@ WARNING: plugin forks a native controller @
This plugin launches a native controller that is not subject to the Java
security manager nor to system call filters.

Continue with installation? [y/N]y
Exception in thread "main" java.nio.file.FileSystemException: /etc/elasticsearch/x-pack/roles.yml: Operation not permitted

(Ioannis Kakavas) #14

Hi again,

Let's take it from the beginning:

  • How did you install Elasticsearch?
  • Since you are using linux, is there any reason why you are not using our .deb or .rpm packages? See here and here.

The elastic user is an internal user that Elasticsearh uses and not an OS user so that would have no effect.

It would probably be easier for you if you stick to one of the supported/suggested ways of installing Elasticsearch ( pick your choice from here) and make sure this works before moving on to x-pack installation. This will not only make it easier for you to do, but also for us to help you with any issues you encounter down the road (since we will know exactly what you have done already and at which point in the process you got stuck by an error )

(Jason Johnson) #15

does this help:

root@oc-elk:~/pkg# ls

root@oc-elk:~/pkg# history | grep elasticsearch-6.2.1.deb
84 dpkg -i elasticsearch-6.2.1.deb

There is something I'm missing when installing x-pack or maybe the user that my elasticsearch cluster is running as (single-node).

(Ioannis Kakavas) #16

Is it safe to assume you followed this guide to the point ?

It's correct that you need to install x-pack with superuser privileges, it's also explicitly mentioned in the documentation. What's strange is that you're getting:

Exception in thread "main" java.nio.file.FileSystemException: /etc/elasticsearch/x-pack/roles.yml: Operation not permitted

You shouldn't be getting this when you run

bin/elasticsearch-plugin install x-pack

as root.

Can you share the output of the commands:

ls -la /etc/elasticsearch/


bin/elasticsearch-plugin install x-pack

(the last one after you have navigated to /etc/elasticsearch with

cd /etc/elasticsearch

(Jason Johnson) #17

checking back on my installation:

I have the repository configured:

I'm running as systemd:
root@oc-elk:/etc/apt/sources.list.d# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-05-02 13:16:09 EDT; 18h ago
Docs: http://www.elastic.co
Main PID: 1153 (java)
Tasks: 123
Memory: 5.5G
CPU: 5h 26min 45.728s
CGroup: /system.slice/elasticsearch.service
└─1153 /usr/bin/java -Xms4g -Xmx4g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+

May 02 13:16:09 oc-elk systemd[1]: Started Elasticsearch.

This is curious:

root@oc-elk:/etc/apt/sources.list.d# curl -X GET "localhost:9200/"
curl: (7) Failed to connect to localhost port 9200: Connection refused

root@oc-elk:/usr/share/elasticsearch# bin/elasticsearch-plugin install x-pack
-> Downloading x-pack from elastic
[=================================================] 100%
@ WARNING: plugin requires additional permissions @

  • java.io.FilePermission \.\pipe* read,write
  • java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries
  • java.lang.RuntimePermission getClassLoader
  • java.lang.RuntimePermission setContextClassLoader
  • java.lang.RuntimePermission setFactory
  • java.net.SocketPermission * connect,accept,resolve
  • java.security.SecurityPermission createPolicy.JavaPolicy
  • java.security.SecurityPermission getPolicy
  • java.security.SecurityPermission putProviderProperty.BC
  • java.security.SecurityPermission setPolicy
  • java.util.PropertyPermission * read,write
    See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
    for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
@ WARNING: plugin forks a native controller @
This plugin launches a native controller that is not subject to the Java
security manager nor to system call filters.

Continue with installation? [y/N]y
Elasticsearch keystore is required by plugin [x-pack-security], creating...
-> Installed x-pack with: x-pack-upgrade,x-pack-graph,x-pack-security,x-pack-watcher,x-pack-monitoring,x-pack-ml,x-pack-logstash,x-pack-deprecation,x-pack-core

root@oc-elk:/usr/share/elasticsearch# ls -la /etc/elasticsearch/
total 44
drwxr-s--- 3 root elasticsearch 4096 May 3 07:53 .
drwxr-xr-x 96 root root 4096 Apr 27 09:15 ..
-rw-rw---- 1 root elasticsearch 418 May 3 07:53 elasticsearch.keystore
-rw-r----- 1 root elasticsearch 2865 Apr 24 14:12 elasticsearch.yml
-rw-rw---- 1 root elasticsearch 2866 Feb 14 08:26 elasticsearch.yml.dpkg-old
-rw-r----- 1 root elasticsearch 2870 Feb 14 08:24 elasticsearch.yml.NEW
-rw-r----- 1 root elasticsearch 2869 Feb 14 08:58 elasticsearch.yml.NEW_621
-rw-rw---- 1 root elasticsearch 2809 Feb 14 12:13 jvm.options
-rw-rw---- 1 root elasticsearch 5091 Nov 10 13:42 log4j2.properties
drwxr-x--- 2 root elasticsearch 4096 May 3 07:53 x-pack

(Jason Johnson) #18

if im installing x-pack as root then following step 6 which is to start elasticsearch i get this output:

root@oc-elk:/usr/share/elasticsearch# bin/elasticsearch
[2018-05-03T08:57:09,524][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [oc-elk] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
... 6 more

(Jason Johnson) #19

I used systemd to restart elasticsearch and here is the next step to configure user passwords:

root@oc-elk:/usr/share/elasticsearch# bin/x-pack/setup-passwords interactive

Connection failure to: failed: Connection refused (Connection refused)

ERROR: Failed to connect to elasticsearch at Is the URL correct and elasticsearch running?

(Ioannis Kakavas) #20

You should not run Elasticsearch as root. We have an explicit bootstrap check for this reason as you already found out.

This is the suggested way to start elasticsearch.

You first try to access it on localhost and then on, is this really where Elasticsearch is listening?
This either means that something is blocking access to port 9200 or that Elasticsearch is not running. You should check to see if elasticsearch started correctly by looking at the logs. You can either check journal entries for errors

sudo journalctl --unit elasticsearch

and/or check your logs at /var/log/elasticsearch.log for indications of what might have gone wrong.

Also, when you run

sudo netstat -nlp | grep 9200


ps aux | grep elasticsearch

what do you get ?

Finally, you can show us your elasticsearch.yml file so that we can see how you have configured elasticsearch.