{"statusCode":404,"error":"Not Found","message":"Not Found"} after security

txmrlevine · October 20, 2020, 7:28pm

after implementing the following lines in my 7.7.1 nodes I try to bring up Kibana and AFTER I use the login credentials I supplied in the password-setup binary I get the 404.

I created the cert on the master node and pushed same cert to all master and data nodes.

        xpack.security.enabled: true
        xpack.security.transport.ssl.enabled: true
        xpack.security.transport.ssl.verification_mode: certificate
        xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
        xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

I am using an http protocol for the kibana stack- am I required to include any of the xpack.http config lines?

anyone responds - needs more info let me know I am new to this.

jportner · October 22, 2020, 1:11pm

You don't need to use SSL/TLS on the Elasticsearch HTTP interface when you enable security (though this is strongly recommended anyway!) Using SSL/TLS on the Transport interface should have no impact on Kibana whatsoever.

Additional info would help here:

What does your Kibana config look like?
Can you share Kibana's log output?
What user are you using to log into Kibana with?

txmrlevine · October 22, 2020, 1:31pm

we normally don't use logging. but I turned it on just for the login itself to see what happened. this is the LOCAL login of the kibana user setup in the password-setup binary.

[root@infralogs-elasticsearchm-101w logs]# tail -f stdout
{"type":"response","@timestamp":"2020-10-19T13:24:29Z","tags":[],"pid":23345,"method":"get","statusCode":304,"req":{"url":"/bundles/plugin/file_upload/file_upload.plugin.js","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"*/*","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"e179509f269543fbaa2e576e42c835f9c5f88a0a-/bundles/plugin/file_upload/-gzip\""},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":304,"responseTime":7,"contentLength":9},"message":"GET /bundles/plugin/file_upload/file_upload.plugin.js 304 7ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:29Z","tags":[],"pid":23345,"method":"get","statusCode":304,"req":{"url":"/bundles/plugin/infra/infra.plugin.js","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"*/*","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"b6c2e4783eaa9a76b510fd497a8e6e0501b3ded1-/bundles/plugin/infra/-gzip\""},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":304,"responseTime":5,"contentLength":9},"message":"GET /bundles/plugin/infra/infra.plugin.js 304 5ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:29Z","tags":[],"pid":23345,"method":"get","statusCode":304,"req":{"url":"/bundles/plugin/dataEnhanced/dataEnhanced.plugin.js","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"*/*","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"ed9c2bb1321500d164717e3eabd5e931c8cc6ec8-/bundles/plugin/dataEnhanced/-gzip\""},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":304,"responseTime":8,"contentLength":9},"message":"GET /bundles/plugin/dataEnhanced/dataEnhanced.plugin.js 304 8ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:29Z","tags":[],"pid":23345,"method":"get","statusCode":304,"req":{"url":"/bundles/plugin/navigation/navigation.plugin.js","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"*/*","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"fcf11bd1055dfd87e5e725a466285ef9a0fd83a9-/bundles/plugin/navigation/-gzip\""},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":304,"responseTime":6,"contentLength":9},"message":"GET /bundles/plugin/navigation/navigation.plugin.js 304 6ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:29Z","tags":[],"pid":23345,"method":"get","statusCode":304,"req":{"url":"/bundles/plugin/graph/graph.plugin.js","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"*/*","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"d8690987f63b137c735ed1b9cdd102b92c1fdb2f-/bundles/plugin/graph/-gzip\""},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":304,"responseTime":4,"contentLength":9},"message":"GET /bundles/plugin/graph/graph.plugin.js 304 4ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:29Z","tags":[],"pid":23345,"method":"get","statusCode":304,"req":{"url":"/bundles/plugin/dashboard/dashboard.plugin.js","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"*/*","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"8de2ce3fa04609a3e0ae5e29c7799c523eda36a9-/bundles/plugin/dashboard/-gzip\""},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":304,"responseTime":4,"contentLength":9},"message":"GET /bundles/plugin/dashboard/dashboard.plugin.js 304 4ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:30Z","tags":[],"pid":23345,"method":"post","statusCode":200,"req":{"url":"/api/core/capabilities","method":"post","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","content-length":"364","kbn-version":"7.7.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://infralogs-prod.active.com:5601","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":200,"responseTime":199,"contentLength":9},"message":"POST /api/core/capabilities 200 199ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:30Z","tags":[],"pid":23345,"method":"get","statusCode":304,"req":{"url":"/bundles/plugin/security/1.plugin.js","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"*/*","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"4b63c3e43c74622206d0e919c24adfa5681a9578-/bundles/plugin/security/-gzip\""},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":304,"responseTime":9,"contentLength":9},"message":"GET /bundles/plugin/security/1.plugin.js 304 9ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:30Z","tags":[],"pid":23345,"method":"get","statusCode":304,"req":{"url":"/bundles/plugin/security/4.plugin.js","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"*/*","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9","if-none-match":"\"807814aec6997877c4546bf19815ad8dcdb98352-/bundles/plugin/security/-gzip\""},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":304,"responseTime":9,"contentLength":9},"message":"GET /bundles/plugin/security/4.plugin.js 304 9ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:31Z","tags":[],"pid":23345,"method":"get","statusCode":200,"req":{"url":"/internal/security/login_state","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","kbn-version":"7.7.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","content-type":"application/json","accept":"*/*","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":200,"responseTime":16,"contentLength":9},"message":"GET /internal/security/login_state 200 16ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:46Z","tags":[],"pid":23345,"method":"post","statusCode":204,"req":{"url":"/internal/security/login","method":"post","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","content-length":"43","kbn-version":"7.7.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://infralogs-prod.active.com:5601","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":204,"responseTime":20,"contentLength":9},"message":"POST /internal/security/login 204 20ms - 9.0B"}
{"type":"response","@timestamp":"2020-10-19T13:24:46Z","tags":[],"pid":23345,"method":"get","statusCode":404,"req":{"url":"/app/kibana","method":"get","headers":{"host":"infralogs-prod.active.com:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.220.224.8","userAgent":"10.220.224.8","referer":"http://infralogs-prod.active.com:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":404,"responseTime":98,"contentLength":9},"message":"GET /app/kibana 404

txmrlevine · October 22, 2020, 1:32pm

my yml file looks like this.

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "removed"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
server.name: "removed"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://removed:9200"]

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
#elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
elasticsearch.username: "kibana"
elasticsearch.password: "removedthis"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
pid.file: /opt/kibana-7.7.1-linux-x86_64/kibana.pid

# Enables you specify a file where Kibana stores log output.
logging.dest: /opt/kibana-7.7.1-linux-x86_64/logs/stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"
################################

BTW - I tried ONLY setting the xpack security setting to True but the node failed - I am using BASIC license says the other protocols are required.

Larry_Gregory · October 22, 2020, 5:17pm

Which user are you attempting to login to Kibana with? Can you run this command against ES directly using those same credentials and post the output?

curl http://elasticsearch-host:9200/_security/_authenticate -u USERNAME:PASSWORD

txmrlevine · October 22, 2020, 5:42pm

elasticsearch will not start if I try to enable -

journalctl -xe
Oct 22 17:39:01 infralogs-elasticsearchm-102w.active.tan polkitd[745]: Registered Authentication Agent
Oct 22 17:39:01 infralogs-elasticsearchm-102w.active.tan systemd[1]: Stopping Elasticsearch...
-- Subject: Unit elasticsearch.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun shutting down.
Oct 22 17:39:01 infralogs-elasticsearchm-102w.active.tan systemd[1]: Stopped Elasticsearch.
-- Subject: Unit elasticsearch.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has finished shutting down.
Oct 22 17:39:01 infralogs-elasticsearchm-102w.active.tan systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd-entrypoint[65585]: ERROR: [1] bootstr
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd-entrypoint[65585]: [1]: Transport SSL
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd-entrypoint[65585]: ERROR: Elasticsear
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd[1]: elasticsearch.service: main proce
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd[1]: Unit elasticsearch.service entere
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd[1]: elasticsearch.service failed.
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan polkitd[745]: Unregistered Authentication Age
lines 2934-2967/2967 (END)

 systemctl -l status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─override.conf
   Active: failed (Result: exit-code) since Thu 2020-10-22 17:39:14 UTC; 1min 50s ago
     Docs: https://www.elastic.co
  Process: 65585 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
 Main PID: 65585 (code=exited, status=78)

Oct 22 17:39:01 infralogs-elasticsearchm-102w.active.tan systemd[1]: Starting Elasticsearch...
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd-entrypoint[65585]: ERROR: [1] bootstrap checks failed
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd-entrypoint[65585]: [1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd-entrypoint[65585]: ERROR: Elasticsearch did not exit normally - check the logs at /prod-infralog/infralogs-elasticsearchm-102w/logs/Entraprise-Logging.log
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd[1]: elasticsearch.service: main process exited, code=exited, status=78/n/a
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd[1]: Failed to start Elasticsearch.
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd[1]: Unit elasticsearch.service entered failed state.
Oct 22 17:39:14 infralogs-elasticsearchm-102w.active.tan systemd[1]: elasticsearch.service failed.

jportner · October 22, 2020, 6:16pm

Sorry, I think there's been some confusion --

Can you:

Re-enable the xpack.security.transport.ssl settings
Start Elasticsearch
Using a terminal, run this curl command against Elasticsearch (using the same username/password you were using to log into Kibana with):

curl http://elasticsearch-host:9200/_security/_authenticate -u USERNAME:PASSWORD

txmrlevine · October 22, 2020, 6:37pm

This is from the host I am running on -

I have no issues sharing the passwd I get up in setup-password because they will be modified when this is fixed.


curl http://<hostname>:9200/_security/_authenticate -u kibana:31kibana

{"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [kibana]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"failed to authenticate user [kibana]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

[root@<host> elasticsearch]#

I know that is the passwd I set.

One question – does the elasticsearch need to be stopped ( on all nodes ) when creating the passwords? I do not know how this is all put together and not sure if I did that correctly.

I have 2 master nodes and 5 data nodes.

jportner · October 22, 2020, 7:47pm

If this password was valid, Elasticsearch should have responded with something like this:

$ curl http://localhost:9200/_security/_authenticate -u kibana:password
{"username":"kibana","roles":["kibana"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true,"authentication_realm":{"name":"reserved","type":"reserved"},"lookup_realm":{"name":"reserved","type":"reserved"},"authentication_type":"realm"}

The response you received indicates that the password was rejected.

Is this the same password that you used in your kibana.yml file?
If so, do you see anything else in Kibana's logs when it starts up?

At any rate, this minimally-privileged user is only for the Kibana server to connect to Elasticsearch. You can't log into Kibana itself with this user (technically you can log in but you won't have the correct privileges to access any Kibana applications, which is why you're seeing the 404 message -- we added an enhancement in 7.10 to improve this experience).

You can instead use the elastic superuser to log into Kibana.

On the "Configuring security in Kibana" page, we mention:

"This must be a user who has been assigned Kibana privileges. Kibana server credentials should only be used internally by the Kibana server."

No, you don't need to stop Elasticsearch when creating/changing passwords for reserved users.

txmrlevine · October 22, 2020, 8:11pm

Thanks – changing to elastic user worked!!

Thanks for your time.

txmrlevine · October 22, 2020, 9:56pm

I will open a new thread if I have to but AFTER I set up the stack and was able to actually login the logstash nodes stopped sending data to the data nodes.

As soon as I backed out the security configuration the logs came in.

DO I have to do something on the logstash nodes?

I saw some info on the site but insure about how to actually set it up.

Something to do with /etc/logstash/config.d/logstash.conf.

Please let me know if I should open a new thread for this.

jportner · October 26, 2020, 7:38pm

You'll need to:

Create a user with the proper privileges to write to your Logstash indices
Configure your Logstash nodes to authenticate with this user

See this page for details: https://www.elastic.co/guide/en/logstash/7.7/ls-security.html
If you run into issues with that I would recommend creating a new topic in the Logstash category so the right people will see it and be able to give you assistance.

txmrlevine · October 26, 2020, 7:51pm

so after I set up and configure security on all other nodes i need to go to kibana build a new user with write access to logstash indices
then

I need to up that logstash.yml "output" section with that username/ passwd combo.

jportner · October 26, 2020, 8:10pm

Yes, that's right!

system · November 23, 2020, 8:10pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.