Not able to login after xpack enabled

[DILIP_SHARMA]

Hi i am running 7.6.1 ES and kibana in Ubuntu server .

once i enabled the xpack security

in ES configuration

xpack.security.enabled: true
discovery.type: single-node

i used following command to generate pwd

bin/elasticsearch-setup-passwords auto

and kibana configuration added following file

elasticsearch.username: "kibana"
elasticsearch.password: "***"

i able to get login screen but not able to login
please find screen -

and i can see following error in syslog

Mar  9 11:43:14 vv-es kibana[10782]: {"type":"response","@timestamp":"2020-03-09T11:43:14Z","tags":[],"pid":10782,
"method":"post","statusCode":404,"req":{"url":"/internal/security/login","method":"post",
"headers":{"x-forwarded-for":"49.206.240.190","x-forwarded-proto":"https","x-forwarded-port":"443","host":"xyz.net",
"x-amzn-trace-id":"Root=1-5e662bd2-a051372e02673d70baaadc8a","content-length":"56","pragma":"no-cache","cache-control":"no-cache","accept":"application/json,
 text/plain, */*","sec-fetch-dest":"empty","kbn-version":"7.6.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64)
 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36","content-type":"application/json;charset=UTF-8",
 "origin":"https://xyz.net","sec-fetch-site":"same-origin","sec-fetch-mode":"cors",
 "referer":"https://xyz.net/login?next=%2Fapp%2Fkibana","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,hi;q=0.8"},
 "remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://xyz.net/login?next=%2Fapp%2Fkibana"}
 ,"res":{"statusCode":404,"responseTime":63,"contentLength":9},"message":"POST /internal/security/login 404 63ms - 9.0B"}

any suggestion in this .
Thanks

[tylersmalley]

It looks like it's getting a 404. If you refresh the page, does the login still come up? Is there some sort of proxy in-front of Kibana?

[DILIP_SHARMA]

yes its give login page even refresh or go different browser .
currently no proxy , i am testing with direct IP which is as good as localhost .

[DILIP_SHARMA]

hi @tylersmalley any suggestion on this ? becuase if i disable xpack , kibana works fine .

and even i tried this

curl -vvv  'http://localhost:5601/mzp/login'  \
-H 'kbn-xsrf: 7.6.1' \
-H 'kbn-version: 7.6.1' \
-H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8'  \
-H 'Accept: application/json, text/javascript, */*; q=0.01'  \
-H 'Connection: keep-alive' \
--data 'username=***&password=***' 

this also throw 404 not found error .

[Larry_Gregory]

Can you provide your full kibana.yml file, redacting any sensitive information?

If we can't find anything there, we may need you to enable debug logging to diagnose further. I haven't come across this particular error before.

[DILIP_SHARMA]

hi please find requested kibana yml

> 
> # Kibana is served by a back end server. This setting specifies the port to use.
> #server.port: 5601
> 
> # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
> # The default is 'localhost', which usually means remote machines will not be able to connect.
> # To allow connections from remote users, set this parameter to a non-loopback address.
> server.host: "0.0.0.0"
> 
> # Enables you to specify a path to mount Kibana at if you are running behind a proxy.
> # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
> # from requests it receives, and to prevent a deprecation warning at startup.
> # This setting cannot end in a slash.
> #server.basePath: ""
> 
> # Specifies whether Kibana should rewrite requests that are prefixed with
> # `server.basePath` or require that they are rewritten by your reverse proxy.
> # This setting was effectively always `false` before Kibana 6.3 and will
> # default to `true` starting in Kibana 7.0.
> #server.rewriteBasePath: false
> 
> # The maximum payload size in bytes for incoming server requests.
> #server.maxPayloadBytes: 1048576
> 
> # The Kibana server's name.  This is used for display purposes.
> #server.name: "your-hostname"
> 
> # The URLs of the Elasticsearch instances to use for all your queries.
> elasticsearch.hosts: ["http://127.0.0.1:9200"]
> 
> # When this setting's value is true Kibana uses the hostname specified in the server.host
> # setting. When the value of this setting is false, Kibana uses the hostname of the host
> # that connects to this Kibana instance.
> #elasticsearch.preserveHost: true
> 
> # Kibana uses an index in Elasticsearch to store saved searches, visualizations and
> # dashboards. Kibana creates a new index if the index doesn't already exist.
> #kibana.index: ".kibana"
> 
> # The default application to load.
> #kibana.defaultAppId: "home"
> 
> # If your Elasticsearch is protected with basic authentication, these settings provide
> # the username and password that the Kibana server uses to perform maintenance on the Kibana
> # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
> # is proxied through the Kibana server.
> elasticsearch.username: "kibana"
> elasticsearch.password: "******"
> 
> # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
> # These settings enable SSL for outgoing requests from the Kibana server to the browser.
> #server.ssl.enabled: false
> #server.ssl.certificate: /path/to/your/server.crt
> #server.ssl.key: /path/to/your/server.key
> 
> # Optional settings that provide the paths to the PEM-format SSL certificate and key files.
> # These files are used to verify the identity of Kibana to Elasticsearch and are required when
> # xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
> #elasticsearch.ssl.certificate: /path/to/your/client.crt
> #elasticsearch.ssl.key: /path/to/your/client.key
> 
> # Optional setting that enables you to specify a path to the PEM file for the certificate
> # authority for your Elasticsearch instance.
> #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
> 
> # To disregard the validity of SSL certificates, change this setting's value to 'none'.
> #elasticsearch.ssl.verificationMode: full
> 
> # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
> # the elasticsearch.requestTimeout setting.
> #elasticsearch.pingTimeout: 1500
> 
> # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
> # must be a positive integer.
> #elasticsearch.requestTimeout: 30000
> 
> # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
> # headers, set this value to [] (an empty list).
> elasticsearch.requestHeadersWhitelist: [ "authorization","securitytenant","sgtenant" ]
> 
> # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
> # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
> #elasticsearch.customHeaders: {}
> 
> # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
> #elasticsearch.shardTimeout: 30000
> 
> # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
> #elasticsearch.startupTimeout: 5000
> 
> # Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
> #elasticsearch.logQueries: false
> 
> # Specifies the path where Kibana creates the process ID file.
> #pid.file: /var/run/kibana.pid
> 
> # Enables you specify a file where Kibana stores log output.
> #logging.dest: /var/log/kibana.log
> 
> # Set the value of this setting to true to suppress all logging output.
> #logging.silent: false
> 
> # Set the value of this setting to true to suppress all logging output other than error messages.
> #logging.quiet: false
> 
> # Set the value of this setting to true to log all events, including system usage information
> # and all requests.
> #logging.verbose: false
> 
> # Set the interval in milliseconds to sample system and process performance
> # metrics. Minimum is 100ms. Defaults to 5000.
> #ops.interval: 5000
> 
> # Specifies locale to be used for all localizable strings, dates and number formats.
> # Supported languages are the following: English - en , by default , Chinese - zh-CN .
> #i18n.locale: "en"

[Larry_Gregory]

Thanks!

@DILIP_SHARMA

elasticsearch.requestHeadersWhitelist: [ "authorization","securitytenant","sgtenant" ]

Do you happen to have SearchGuard installed, or are you running OpenDistro for Elasticsearch?
sgtenant and securitytenant both look like SearchGuard headers to me, but I'm honestly not too familiar with their setup.

In any event, if you are running either of these, we unfortunately don't offer support for them, and you should contact either SearchGuard or OpenDistro.

If you're not running either of these, then you shouldn't have to whitelist the authorization header either. If you enable debug logging, do the logs show anything interesting that could help? (set logging.verbose: true in your kibana.yml to enable this)

[DILIP_SHARMA]

i not using any of those , i just trying based on forums , i will disable that
other thing which i see
my local machine where its works
request url

/api/security/v1/login

and server where its not working
internal/security/login

anything to do with that ?
and i disabled the
whitelist header

also let me enable log and check

[DILIP_SHARMA]

please find after enable logging.verbose: true

t/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36","content-type":"application/json;charset=UTF-8","origin":"http://******:5601","referer":"http://******:5601/login?next=%2F","accept-encoding":"gzip, deflate","accept-language":"en-GB,en-US;q=0.9,en;q=0.8"},"remoteAddress":"3.0.63.253","userAgent":"3.0.63.253","referer":"http://******:5601/login?next=%2F"},"res":{"statusCode":404,"responseTime":66,"contentLength":9},"message":"POST /internal/security/login 404 66ms - 9.0B"}
Mar 10 13:41:05 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:05Z","tags":["debug","plugins","usageCollection"],"pid":15402,"message":"Fetching data from kibana_stats collector"}
Mar 10 13:41:05 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:05Z","tags":["debug","plugins","usageCollection"],"pid":15402,"message":"Fetching data from kibana_settings collector"}
Mar 10 13:41:05 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:05Z","tags":["debug","plugins","usageCollection"],"pid":15402,"message":"not sending [kibana_settings] monitoring document because [undefined] is null or invalid."}
Mar 10 13:41:05 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:05Z","tags":["debug","monitoring","kibana-monitoring"],"pid":15402,"message":"Uploading bulk stats payload to the local cluster"}
Mar 10 13:41:05 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:05Z","tags":["debug","monitoring","kibana-monitoring"],"pid":15402,"message":"Uploaded bulk stats payload to the local cluster"}
Mar 10 13:41:05 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:05Z","tags":["debug","monitoring","kibana-monitoring"],"pid":15402,"message":"Received Kibana Ops event data"}
Mar 10 13:41:05 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:05Z","tags":["debug","monitoring","kibana-monitoring"],"pid":15402,"message":"Received Kibana Ops event data"}
Mar 10 13:41:05 ekfc-es kibana[15402]: {"type":"response","@timestamp":"2020-03-10T13:41:05Z","tags":[],"pid":15402,"method":"get","statusCode":200,"req":{"url":"/login","method":"get","headers":{"host":"******","connection":"close","user-agent":"ELB-HealthChecker/2.0","accept-encoding":"gzip, compressed"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":200,"responseTime":21,"contentLength":9},"message":"GET /login 200 21ms - 9.0B"}
Mar 10 13:41:06 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:06Z","tags":["debug","monitoring","kibana-monitoring"],"pid":15402,"message":"Received Kibana Ops event data"}
Mar 10 13:41:06 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:06Z","tags":["debug","monitoring","kibana-monitoring"],"pid":15402,"message":"Received Kibana Ops event data"}
Mar 10 13:41:07 ekfc-es kibana[15402]: {"type":"ops","@timestamp":"2020-03-10T13:41:07Z","tags":[],"pid":15402,"os":{"load":[0.052734375,0.14892578125,0.27099609375],"mem":{"total":4086702080,"free":557871104},"uptime":70637},"proc":{"uptime":340.294,"mem":{"rss":401289216,"heapTotal":319270912,"heapUsed":245207320,"external":2377003},"delay":0.1028829962015152},"load":{"requests":{"5601":{"total":2,"disconnects":0,"statusCodes":{"200":1,"404":1}}},"responseTimes":{"5601":{"avg":43.5,"max":66}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 233.8MB uptime: 0:05:40 load: [0.05 0.15 0.27] delay: 0.103"}
Mar 10 13:41:10 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:10Z","tags":["debug","monitoring","kibana-monitoring"],"pid":15402,"message":"Received Kibana Ops event data"}
Mar 10 13:41:10 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:10Z","tags":["debug","monitoring","kibana-monitoring"],"pid":15402,"message":"Received Kibana Ops event data"}
Mar 10 13:41:10 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:10Z","tags":["debug","http","server","Kibana","cookie-session-storage"],"pid":15402,"message":"Error: Unauthorized"}
Mar 10 13:41:10 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:10Z","tags":["debug","plugins","security","basic"],"pid":15402,"message":"Trying to authenticate user request to /."}
Mar 10 13:41:10 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:10Z","tags":["debug","plugins","security","basic"],"pid":15402,"message":"Trying to authenticate via header."}
Mar 10 13:41:10 ekfc-es kibana[15402]: {"type":"log","@timestamp":"2020-03-10T13:41:10Z","tags":["debug","plugins","security","basic"],"pid":15402,"message":"Authorization header is not presented."}
Mar 10 13:41:10 ekfc-es kibana[15402]: {"type":"response","@timestamp":"2020-03-10T13:41:10Z","tags":[],"pid":15402,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"******","connection":"close","user-agent":"ELB-HealthChecker/2.0","accept-encoding":"gzip, compressed"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":302,"responseTime":9,"contentLength":9},"message":"GET / 302 9ms - 9.0B"}