Elasticsearch stop working after enable x-pack

Elastic Stack Elasticsearch
1

I want kibana to ask for the password every time i use it, so i came up with this x-pack thing and tried to enable x-pack in elasticsearch, my bin folder haver x-pack-env, x-pack-security-env, x-pack-watcher-env i did changes in yml file

elasticsearch.yml

#
# Use a descriptive name for your cluster:
#
cluster.name: ElasticsearchStaging
#xpack.security.enabled: true
#

after changing it stop working. this is my log file:

elasticsearch.log

Aug 13 05:10:15 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 05:10:27 elastic-stage-vyakar elasticsearch[947]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed i
Aug 13 05:25:39 elastic-stage-vyakar systemd[1]: Stopping Elasticsearch...
Aug 13 05:25:40 elastic-stage-vyakar systemd[1]: Stopped Elasticsearch.
Aug 13 05:25:40 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 05:25:41 elastic-stage-vyakar elasticsearch[2719]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed 
Aug 13 05:26:08 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Aug 13 05:26:08 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Aug 13 05:27:39 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 05:27:40 elastic-stage-vyakar elasticsearch[2894]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

Thanks.

2

kibana.yml

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana"
elasticsearch.password: "pass"
3

Hi,

This is unfortunately not enough for us to help you. Please tell us what version you are using, how did you install X-Pack and give us the full configuration. For example, your elasticsearch.yml file you shared, only contains an extra line and it is commented out so it is not in effect.

Do note that security for elasticsearch is free ( part of the Basic license) in recent versions , see https://www.elastic.co/blog/security-for-elasticsearch-is-now-free . You might want to use this instead of attempting to install X-Pack in an older version of Elasticsearch

4

First of all, thank you for the reply, and sorry for not enough information. So i have elasticsearch and kibana version 7.3 and x-pack is there is elasticsearch by default

this is the link i used to install elasticsearch :
https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

this is i used for kibana:
https://www.elastic.co/guide/en/kibana/current/deb.html

under this section Download and install the Debian package manually

elasticsearch.yml :

#
# Use a descriptive name for your cluster:
#
cluster.name: ElasticsearchStaging
#xpack.security.enabled: true
#xpack.security.audit.enabled: true
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 10.***.**.**
#
# Set a custom port for HTTP:
#
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: 10.***.**.**
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: "node-1"
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

i commented xpack.security.enabled: true because it was giving error

5

kibana.yml :

server.port: 5601
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: 10.***.*.**
# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""
# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
# The Kibana server's name.  This is used for display purposes.
server.name: testing
# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: http://10.***.*.**:9200
# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: kibanaindx
# The default application to load.
#kibana.defaultAppId: "home"
# The Kibana server's name.  This is used for display purposes.
server.name: testing
# The default application to load.
#kibana.defaultAppId: "home"
# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana"
elasticsearch.password: "pass"
# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
# Optional setting that enables you to specify a path to the PEM file for the certificate
6

You'll have to read through the documentation. It will help you understand what needs to be configured and why and if you get stack at a specific point, then we will be able to assist you further when you have specific questions or issues. I suggest you start reading from here : https://www.elastic.co/guide/en/elastic-stack-overview/current/elasticsearch-security.html and follow the links according to which security features you want to enable

7

Yes, i have been trying this very long but still getting error even after following the steps.
https://www.elastic.co/guide/en/elastic-stack-overview/current/get-started-enable-security.html
i follow above link to enable security features in elasticsearch but still got error.

8

Apologies, but this is hardly enough to get us started with helping you.

You need to share exactly what you have done already and what exactly errors you are getting. Otherwise it is really hard for anyone to even start looking into assisting you.

9

No i am sorry that i am not able to give you information that you want, But documentation is confusing me it have lots of thing, tell me one thing is LDAP is important if want to set password for kibana, i just want one thing kibana should ask for password and user id when i access it from my browser. Now let me explain you what i tried yet.
That link i mentioned, it ask to use kibana with x-pack 'OK no prob' so the first step is to configure security in elasticsearch

so this link ask me to enable xpack.secuiry.enabled: true that i did in my elasticsearch.yml file but after doing that my elasticsearch is stop working now tell me where am i wrong.
Note: I didn't buy any license, i install both elasticsearch and kibana for free.

is that the reason i am getting an error after enabling xpack.security.enabled
but i think even the free elasticsearch and kibana have the feature to activate password for kibana, if they do then, please suggest me what should i do or which link do i have to follow, but all the sites have very complex things, i am just getting confused what thing do i have to follow. Is there any information that you want ? if there anything then please ask

Thanks

1 Like
10

What do you mean "stop working" , what is the error, what do you see in your logs ?

11

I should really thank you, for the reply. :slightly_smiling_face:

the command i use for logs journalctl -u elsticsearch.service
this is log:

Aug 13 11:26:47 elastic-stage-vyakar systemd[1]: Stopping Elasticsearch...
Aug 13 11:26:47 elastic-stage-vyakar systemd[1]: Stopped Elasticsearch.
Aug 13 11:26:47 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 11:26:49 elastic-stage-vyakar elasticsearch[2625]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Aug 13 11:27:14 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Aug 13 11:27:14 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

after using service elasticsearch status

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-08-13 11:27:14 UTC; 3min 22s ago
     Docs: http://www.elastic.co
  Process: 2625 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
 Main PID: 2625 (code=exited, status=78)
Aug 13 11:26:47 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 11:26:49 elastic-stage-vyakar elasticsearch[2625]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed 
Aug 13 11:27:14 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Aug 13 11:27:14 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

after commenting xpack.security.enabled elasticsearch comes back to running state

i also tried to set password to elasticsearch by using this command :

bin/elasticsearch-setup-passwords interactive

but it gives me this:

It doesn't look like the X-Pack security feature is enabled on this Elasticsearch node.
Please check if you have enabled X-Pack security in your elasticsearch.yml configuration file.
12

Did you read this article? https://www.elastic.co/blog/configuring-ssl-tls-and-https-to-secure-elasticsearch-kibana-beats-and-logstash

I think you also need to enable the certificate authentication and provide the proper certificates...

13

Let's solve the issues, one at a time. If you can't successfully configure xpack.security.enabled, nothing more will work after that.

There should be an elasticsearch.log file in /var/log/elasticsearch/ , can you share the contents here with us ? This will contain the information regarding why Elasticsearch fails to start.

14

Yes sure,

tail -30 /var/log/elasticsearch/elasticsearch.log

[2019-08-12T11:07:58,872][INFO ][o.e.c.c.ClusterBootstrapService] [elastic-stage-vyakar] no discovery configuration found, will perform best-effort cluster bootstrapping after
 [3s] unless existing master is discovered
[2019-08-12T11:08:01,878][INFO ][o.e.c.c.Coordinator      ] [elastic-stage-vyakar] setting initial configuration to VotingConfiguration{eytGM4YaSh6r2vvbXRObxQ}
[2019-08-12T11:08:02,073][INFO ][o.e.c.s.MasterService    ] [elastic-stage-vyakar] elected-as-master ([1] nodes joined)[{elastic-stage-vyakar}{eytGM4YaSh6r2vvbXRObxQ}{uwTrrkue
Tiq44Z9j4xGQhg}{127.0.0.1}{127.0.0.1:9300}{dim}{ml.machine_memory=3872571392, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_]
, term: 1, version: 1, reason: master node changed {previous [], current [{elastic-stage-vyakar}{eytGM4YaSh6r2vvbXRObxQ}{uwTrrkueTiq44Z9j4xGQhg}{127.0.0.1}{127.0.0.1:9300}{dim
}{ml.machine_memory=3872571392, xpack.installed=true, ml.max_open_jobs=20}]}
[2019-08-12T11:08:02,138][INFO ][o.e.c.c.CoordinationState] [elastic-stage-vyakar] cluster UUID set to [mnIhuTX_QH6KUAA9imZjLw]
[2019-08-12T11:08:02,167][INFO ][o.e.c.s.ClusterApplierService] [elastic-stage-vyakar] master node changed {previous [], current [{elastic-stage-vyakar}{eytGM4YaSh6r2vvbXRObxQ
}{uwTrrkueTiq44Z9j4xGQhg}{127.0.0.1}{127.0.0.1:9300}{dim}{ml.machine_memory=3872571392, xpack.installed=true, ml.max_open_jobs=20}]}, term: 1, version: 1, reason: Publication{
term=1, version=1}
[2019-08-12T11:08:02,288][INFO ][o.e.h.AbstractHttpServerTransport] [elastic-stage-vyakar] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2019-08-12T11:08:02,288][INFO ][o.e.n.Node               ] [elastic-stage-vyakar] started
[2019-08-12T11:08:02,477][INFO ][o.e.g.GatewayService     ] [elastic-stage-vyakar] recovered [0] indices into cluster_state
[2019-08-12T11:08:03,080][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elastic-stage-vyakar] adding template [.watch-history-10] for index patterns [.watcher-history-10*]
[2019-08-12T11:08:03,153][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elastic-stage-vyakar] adding template [.watches] for index patterns [.watches*]
[2019-08-12T11:08:03,216][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elastic-stage-vyakar] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2019-08-12T11:08:03,291][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elastic-stage-vyakar] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7
-*]
[2019-08-12T11:08:03,410][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elastic-stage-vyakar] adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]
[2019-08-12T11:08:03,524][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elastic-stage-vyakar] adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]
[2019-08-12T11:08:03,585][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elastic-stage-vyakar] adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]
[2019-08-12T11:08:03,682][INFO ][o.e.c.m.MetaDataIndexTemplateService] [elastic-stage-vyakar] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]
[2019-08-12T11:08:03,741][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [elastic-stage-vyakar] adding index lifecycle policy [watch-history-ilm-policy]
[2019-08-12T11:08:04,031][INFO ][o.e.l.LicenseService     ] [elastic-stage-vyakar] license [87580efd-703f-4521-8736-377cba95a277] mode [basic] - valid
[2019-08-12T11:08:04,039][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [elastic-stage-vyakar] Active license is now [BASIC]; Security is disabled
[2019-08-12T11:11:54,839][INFO ][o.e.n.Node               ] [elastic-stage-vyakar] stopping ...
[2019-08-12T11:11:54,855][INFO ][o.e.x.w.WatcherService   ] [elastic-stage-vyakar] stopping watch service, reason [shutdown initiated]
[2019-08-12T11:11:54,995][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elastic-stage-vyakar] [controller/3033] [Main.cc@150] Ml controller exiting
[2019-08-12T11:11:55,002][INFO ][o.e.x.m.p.NativeController] [elastic-stage-vyakar] Native controller process has stopped - no new native processes can be started
[2019-08-12T11:11:55,020][INFO ][o.e.n.Node               ] [elastic-stage-vyakar] stopped
[2019-08-12T11:11:55,021][INFO ][o.e.n.Node               ] [elastic-stage-vyakar] closing ...
[2019-08-12T11:11:55,067][INFO ][o.e.n.Node               ] [elastic-stage-vyakar] closed

but the thing is i don't think it getting updated if i look at the time or date. That's why i was using command journalctl -u elsticsearch.service

15

It does look like it's from yesterday.

Do you have leftover files from a previous installation or is this a clean new one? It could be worth deleting/uninstalling everything and starting from scratch since you don't have any data in there either way.

You can see how you can enable your logs to show in the systemd journal in our docs: Install Elasticsearch with Debian Package | Elasticsearch Guide [8.11] | Elastic

16

Is it really important to reinstall elasticsearch and kibana because yesterday only i did that. Or is there any other way to update log files

17

I can't answer this question with certainty with the information I have, it was a suggestion. I linked you to our docs in my previous message which detail how to enable logs to show in systemd journal. Do that and then take a look at the logs there to see what goes wrong when you set

xpack.secuiry.enabled: true
18

Hello its seems like i have to reinstall elasticsearch, but tell me one thing do i also need to uninstall kibana??

19

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the Community Code of Conduct specifically the section "Be patient".

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

Why is that ? Did you try to show logs in systemd journal and that failed somehow ? It was a suggestion, as this will take you a couple of minutes, instead of trying to salvage an installation if you're not sure what you have changed etc. but it's not necessary . You can still probably make it work with what you have.

I don;t think so, the only problems I've seen so far is that you can't start Elasticsearch with security enabled, not sure Kibana has anything to do with this.

1 Like
20

I am really sorry i am behaving like a dumb but i really found it so hard even if i know its very basic thing, but again
I tried journalctl -f

Aug 13 16:51:26 elastic-stage-vyakar systemd[1]: Stopping Elasticsearch...
Aug 13 16:51:26 elastic-stage-vyakar systemd[1]: Stopped Elasticsearch.
Aug 13 16:51:26 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 16:51:28 elastic-stage-vyakar elasticsearch[2192]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed 
in a future release.
Aug 13 16:51:51 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Aug 13 16:51:51 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Aug 13 16:52:53 elastic-stage-vyakar systemd[1]: Starting Cleanup of Temporary Directories...
Aug 13 16:52:53 elastic-stage-vyakar systemd[1]: Started Cleanup of Temporary Directories.
Aug 13 16:57:49 elastic-stage-vyakar sshd[2436]: error: Received disconnect from 117.222.64.242 port 63055:14: No supported authentication methods available [preauth]
Aug 13 16:57:49 elastic-stage-vyakar sshd[2436]: Disconnected from authenticating user root 117.222.64.242 port 63055 [preauth]

journalctl --unit elasticsearch

Aug 13 16:37:51 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 16:38:01 elastic-stage-vyakar elasticsearch[890]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed i
Aug 13 16:51:26 elastic-stage-vyakar systemd[1]: Stopping Elasticsearch...
Aug 13 16:51:26 elastic-stage-vyakar systemd[1]: Stopped Elasticsearch.
Aug 13 16:51:26 elastic-stage-vyakar systemd[1]: Started Elasticsearch.
Aug 13 16:51:28 elastic-stage-vyakar elasticsearch[2192]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed 
Aug 13 16:51:51 elastic-stage-vyakar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Aug 13 16:51:51 elastic-stage-vyakar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.