"}
[2019-08-25T13:33:13,820][INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-08-25T13:33:13,875][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"main"}
[2019-08-25T13:33:14,006][INFO][logstash.inputs.udp] Starting UDP listener {:address=>"0.0.0.0:5045"}
[2019-08-25T13:33:14,032][INFO][logstash.inputs.udp] Starting UDP listener {:address=>"0.0.0.0:5047"}
[2019-08-25T13:33:14,190][INFO][org.logstash.beats.Server] Starting server on port: 5044
[2019-08-25T13:33:14,222][INFO][logstash.inputs.udp] UDP listener started {:address=>"0.0.0.0:5047", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2019-08-25T13:33:14,232][INFO][logstash.inputs.udp] UDP listener started {:address=>"0.0.0.0:5045", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2019-08-25T13:33:14,269][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-08-25T13:33:14,727][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2019-08-25T13:33:37,297][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://localhost:9200/_xpack'"}
[2019-08-25T13:34:07,221][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://localhost:9200/_xpack'"}
```
I also tried adding the x\_pwd to the keystore and setting
`xpack.monitoring.elasticsearch.password: $x_pwd`
Same thing
Thanks for your help
---
### Author:  [@mbelaloui](https://discuss.elastic.co/u/mbelaloui)
#### Post date: [August 25, 2019, 2:49pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/6 "2019-08-25T14:49:13Z")
Hello @VamPikmin,
try this :
`bin/logstash-keystore add xpack.monitoring.elasticsearch.password`
and then enter your password word.
NB : you can comment/delete this line in the .yml
Mehdi.
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [August 25, 2019, 9:59pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/7 "2019-08-25T21:59:52Z")
Hi @mbelaloui
Thanks for your help
I've added it to the keystore but still get 401
```
/usr/share/logstash/bin/logstash-keystore list --path.settings /etc/logstash
es_pwd
x_pwd
xpack.monitoring.elasticsearch.password
```
Just want to confirm you want me to comment out the pass from the logstash.yml like this:
```
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
#xpack.monitoring.elasticsearch.password: mytextpass
xpack.monitoring.elasticsearch.hosts: ["https://localhost:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/certs/ca.pem
```
In Kibana it was easy
`sudo -u kibana /usr/share/kibana/bin/kibana-keystore add elasticsearch.password`
kibana.yml
```
elasticsearch.username: "kibana"
elasticsearch.password: "elasticsearch.password"
```
---
### Author:  [@mbelaloui](https://discuss.elastic.co/u/mbelaloui)
#### Post date: [August 26, 2019, 10:55am UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/8 "2019-08-26T10:55:36Z")
Hi @VamPikmin
Did it work before adding the keystore rule ???
If no, i think that your user dos'nt have the right to monitore data, to do so you have to add the monitoring rols to your logstash\_system.
If not it certainly came from the https, you don't use the right Ca or the user that run logstash dosn't have the correct access right to read it.
Happy to help you Mehdi.
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [August 26, 2019, 12:12pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/9 "2019-08-26T12:12:37Z")
Hi @mbelaloui
It works with the logstash\_system user and plain text password in config, that should mean that the user has the right access I would imagine?
Thank you 🙂
---
### Author:  [@mbelaloui](https://discuss.elastic.co/u/mbelaloui)
#### Post date: [August 26, 2019, 1:45pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/10 "2019-08-26T13:45:28Z")
> [@VamPikmin](#):
>
> In Kibana it was easy
>
> `sudo -u kibana /usr/share/kibana/bin/kibana-keystore add elasticsearch.password`
>
> kibana.yml
>
> ```auto
> elasticsearch.username: "kibana"
> elasticsearch.password: "elasticsearch.password"
>
> ```
Does it work for you in kibana ?
I just tryed it with kibana, elasticsearch and logstash 7.2.0 and it's ok for me.
My logstash config:
```
cat config/logstash.yml | grep "^[^#]"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: "Remote_beats"
xpack.monitoring.elasticsearch.hosts: ["elasticsearch-node-1:9200", "elasticsearch-node-2:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/home/elastic-user/logstash/ca.crt"
```
In logstash-Keystore
```
bin/logstash-keystore list
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.util.SecurityHelper to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of org.jruby.util.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
xpack.monitoring.elasticsearch.password
```
My kibana config
```
cat /etc/kibana/kibana.yml | grep ^[^#]
server.host: "kibana"
logging.dest: /var/log/kibana/kibana.log
elasticsearch.hosts: ["https://elasticsearch-node-1:9200", "https://elasticsearch-node-2:9200", "https://DESKTOP-JQ2FTSG:9200"]
xpack.monitoring.enabled: true
elasticsearch.username: "kibana"
server.ssl.enabled: true
server.ssl.key: /etc/kibana/kibana-web-cert/kibana-node.key
server.ssl.certificate: /etc/kibana/kibana-web-cert/kibana-node.crt
csp.strict: true
xpack.security.enabled: true
elasticsearch.ssl.certificateAuthorities: /etc/kibana/ca/ca.crt
```
in Kibana-keystor
```
sudo /usr/share/kibana/bin/kibana-keystore --allow-root list
elasticsearch.password
server.ssl.keyPassphrase
```
then, it's ok when i try this :
`bin/logstash -e 'input { stdin { } } output { stdout {} }`
and also i'am seeing the logstash in the monitoring kibana interface.
NB: when you add your password to the keystore be sure that you set your password correctly and without the \<""\>.
Mehdi.
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [August 26, 2019, 10:25pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/11 "2019-08-26T22:25:54Z")
Kibana is fine
This is what my config looks like:
```
cat /etc/kibana/kibana.yml | grep ^[^#]
server.host: "localhost"
elasticsearch.hosts: ["https://localhost:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "elasticsearch.password"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/instance.crt
server.ssl.key: /etc/kibana/certs/instance.key
elasticsearch.ssl.certificate: /etc/kibana/certs/instance.crt
elasticsearch.ssl.key: /etc/kibana/certs/instance.key
elasticsearch.ssl.certificateAuthorities: /etc/kibana/certs/ca.pem
elasticsearch.ssl.verificationMode: certificate
```
Since then I also added this to the kibana config to match yours
```
xpack.monitoring.enabled: true
xpack.security.enabled: true
```
This is all I have in kibana keystore as shown before
```
/usr/share/kibana/bin/kibana-keystore --allow-root list
elasticsearch.password
```
My logstash.yml
```
cat /etc/logstash/logstash.yml | grep ^[^#]
node.name: elastic18
path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.hosts: ["https://localhost:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/certs/ca.pem
```
I always get this error when I run logstash-keystore but not trying to worry about that now
```
/usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list
2019-08-27 07:52:03,668 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2019-08-27 07:52:03,668 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2019-08-27 07:52:03,669 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2019-08-27 07:52:03,669 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
xpack.monitoring.elasticsearch.password
```
I recreated the logstash-keystore again and made sure the password is correct and without double quotes but still keep getting 401
```
curl -XGET https://localhost:9200/_xpack/security/_authenticate?pretty -u logstash_system -k
Enter host password for user 'logstash_system':
{
"username" : "logstash_system",
"roles" : [
"logstash_system"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true,
"authentication_realm" : {
"name" : "reserved",
"type" : "reserved"
},
"lookup_realm" : {
"name" : "reserved",
"type" : "reserved"
}
}
```
I'm probably doing something stupid just don't know what
Thanks for your help
---
### Author:  [@mbelaloui](https://discuss.elastic.co/u/mbelaloui)
#### Post date: [August 27, 2019, 9:19am UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/12 "2019-08-27T09:19:04Z")
Hello @VamPikmin,
I think that this error can be related to your problems.
```auto
/usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list
2019-08-27 07:52:03,668 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2019-08-27 07:52:03,668 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2019-08-27 07:52:03,669 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2019-08-27 07:52:03,669 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
xpack.monitoring.elasticsearch.password
```
Try to kill all the instances of logstash that are running, and be sur that :
`LOGSTASH_HOME=/usr/share/logstash`
`LOGSTASH_CONF=/etc/logstash`
And tell me if it's works.
Very happy to help !!!
Mehdi.
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [August 27, 2019, 9:57am UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/13 "2019-08-27T09:57:55Z")
Hi Mehdi,
Thanks for your help, I really appreciate it.
Even when I export the above settings I still get the error
```
/usr/share/logstash/bin/logstash-keystore list
2019-08-27 19:49:25,498 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2019-08-27 19:49:25,499 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2019-08-27 19:49:25,499 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2019-08-27 19:49:25,499 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
```
My logstash.yml and logstash,keystore are in /etc/logstash
I tried to export LS\_HOME but the error is still present
```
/usr/share/logstash/bin/logstash-keystore list
2019-08-27 19:51:36,188 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2019-08-27 19:51:36,188 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2019-08-27 19:51:36,189 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2019-08-27 19:51:36,189 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
root@elastic18:/etc/logstash# echo $LS_HOME
/etc/logstash/config
```
When I specify the path manually it shows the entry
```
root@elastic18:/etc/logstash# /usr/share/logstash/bin/logstash-keystore list --path.settings /etc/logstash/
2019-08-27 19:53:21,296 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2019-08-27 19:53:21,297 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2019-08-27 19:53:21,297 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2019-08-27 19:53:21,298 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
```
When I copy both logstash.yml and logstash.keystore to /etc/logstash/config it doesn't print the xpack.monitoring.elasticsearch.password entry
I'm running Ubuntu Server 18.04 and logstash service is stopped like you advised
What are your file permissions?
```
/etc/logstash# ll
total 5672
drwxrwxr-x 6 root root 4096 Aug 27 19:48 ./
drwxr-xr-x 100 root root 4096 Aug 23 06:44 ../
drwxr-xr-x 2 root root 4096 Aug 22 18:36 certs/
drwxrwxr-x 2 root root 4096 Aug 24 19:22 conf.d/
drwxr-xr-x 2 root root 4096 Aug 27 19:49 config/
-rw-r--r-- 1 root root 5747716 Aug 22 08:04 GeoLite2-ASN.mmdb
-rw-r--r-- 1 root root 1915 Jul 25 05:55 jvm.options
-rw-r--r-- 1 root root 4987 Jun 21 03:26 log4j2.properties
-rw-r--r-- 1 root root 1428 Aug 27 20:02 logstash.keystore
-rw-r--r-- 1 root root 8578 Aug 27 08:46 logstash.yml
-rw-r--r-- 1 root root 285 Jun 21 03:26 pipelines.yml
-rw------- 1 root root 1696 Jun 21 03:26 startup.options
drwxr-xr-x 2 root root 4096 Aug 24 19:14 '${sys:ls.logs}'/
```
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [August 27, 2019, 10:06am UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/14 "2019-08-27T10:06:24Z")
I've found the error thrown when running logstash-keystore is because I am running from /etc/logstash
If I change the directory to /root for example I don't get it
>
I've redone the keystore again while running from /root directory and the error 401 shows in logstash, so the errors I got above might not be the reason why it's not working
Thank you 🙂
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [August 28, 2019, 2:56am UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/15 "2019-08-28T02:56:13Z")
With both username and password specified in the logstash.yml a successful start:
```
[2019-08-28T12:38:53,514][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.3.1"}
[2019-08-28T12:38:56,062][INFO][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2019-08-28T12:38:56,065][INFO][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2019-08-28T12:38:58,050][INFO][org.reflections.Reflections] Reflections took 48 ms to scan 1 urls, producing 19 keys and 39 values
[2019-08-28T12:38:58,760][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@localhost:9200/]}}
```
If I specify only the username in the logstash.yml and the password in the keystore as xpack.monitoring.elasticsearch.password
```
[2019-08-28T12:37:11,271][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.3.1"}
[2019-08-28T12:37:13,931][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://localhost:9200/_xpack'"}
[2019-08-28T12:37:14,058][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
```
If I intentionally comment out both the username and password in logstash.yml I get the same error as above which leads me to believe that either both settings or none are required.
There is nothing I can find in the documentation that allows to use a keystore instead of a plain text password
EDIT:
There seems to be an issue with this line in logstash.yml
`xpack.monitoring.elasticsearch.hosts: ["https://localhost:9200"]`
I've tried using 127.0.0.1, no joy
If I change to 0.0.0.0 I get a certificate error, so I would need to regenerate them I guess
`[2019-08-28T14:27:50,973][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Host name '0.0.0.0' does not match the certificate subject provided by the peer (CN=localhost)"}`
The main problem is definitely the inability to use just the username and keystore, both need to be specified for this to work 😕
`[2019-08-28T14:38:35,698][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:xxxxxx@localhost:9200/]}}`
---
### Author:  [@mbelaloui](https://discuss.elastic.co/u/mbelaloui)
#### Post date: [August 28, 2019, 1:41pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/16 "2019-08-28T13:41:59Z")
Hi @VamPikmin,
Can you tell me more about your Os, java version and ELK version.
Mehdi.
---
### Author:  [@mbelaloui](https://discuss.elastic.co/u/mbelaloui)
#### Post date: [August 28, 2019, 1:44pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/17 "2019-08-28T13:44:03Z")
I think that logstash can work without java, but keystore no
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [August 28, 2019, 10:09pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/18 "2019-08-28T22:09:44Z")
Hi @mbelaloui
ELK 7.3.1
Ubuntu 18.04.3 LTS
java -version
openjdk version "1.8.0\_222"
OpenJDK Runtime Environment (build 1.8.0\_222-8u222-b10-1ubuntu1~18.04.1-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
You say it's working for you with just the username-
xpack.monitoring.elasticsearch.username: "Remote\_beats"
Maybe I should change to using network ip instead of localhost and see whether that works
EDIT: Nope 😕
`[ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'https://192.168.131.140:9200/_xpack'"}`
---
### Author:  [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [August 28, 2019, 11:38pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/19 "2019-08-28T23:38:12Z")
Well, in the end it was an issue with the ES\_PWD. I exported it as root logstash runs under the logstash account and I can't su to logstash to export it under the right account
I added LOGSTASH\_KEYSTORE\_PASS to
```
cat /etc/default/logstash
LS_HOME="/usr/share/logstash"
LS_SETTINGS_DIR="/etc/logstash"
LS_PIDFILE="/var/run/logstash.pid"
LS_USER="logstash"
LS_GROUP="logstash"
LS_GC_LOG_FILE="/var/log/logstash/gc.log"
LS_OPEN_FILES="16384"
LS_NICE="19"
SERVICE_NAME="logstash"
SERVICE_DESCRIPTION="logstash"
LOGSTASH_KEYSTORE_PASS="myplaintextpass"
```
After that I was able to use anything from the keystore as the password
```
/usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash/ list
elasticsearch.password
es_pwd
xpack.monitoring.elasticsearch.password
```
So either of these will work:
```
xpack.monitoring.elasticsearch.password: "${ES_PWD}"
xpack.monitoring.elasticsearch.password: "${elasticsearch.password}"
xpack.monitoring.elasticsearch.password: "${xpack.monitoring.elasticsearch.password}"
```
The ES\_PWD variable will also work in logstash.output.elasticsearch which is great
Thanks for all your help @mbelaloui and @rugenl
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 25, 2019, 11:38pm UTC](https://discuss.elastic.co/t/logstash-xpack-monitoring-elasticsearch-password/196604/20 "2019-09-25T23:38:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.