Kibana+elasticsearch with authentication from elastic git repository fails

Hello you all,
Im trying to configure kibana with elasticsearch using the xpack security module from the elastic helm official repository (https://github.com/elastic/helm-charts)

I followed the guides for the security implementation using these values files:

elasticsearch_values.yaml

(...)

esConfig:
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificate.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificate.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificate.p12
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificate.p12

extraEnvs:

  • name: ELASTIC_PASSWORD
    valueFrom:
    secretKeyRef:
    name: elastic-credentials
    key: password
  • name: ELASTIC_USERNAME
    valueFrom:
    secretKeyRef:
    name: elastic-credentials
    key: username

secretMounts:

  • name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.1.1"
imagePullPolicy: "IfNotPresent"

(...)

and

kibana_values.yaml

elasticsearchURL: "" # "http://elasticsearch-master:9200"
elasticsearchHosts: "https://elasticsearch-master:9200"

replicas: 1

extraEnvs:

  • name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
    secretKeyRef:
    name: elastic-credentials
    key: username
  • name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
    secretKeyRef:
    name: elastic-credentials
    key: password

secretMounts:

  • name: elastic-certificate-pem
    secretName: elastic-certificate-pem
    path: /usr/share/kibana/config/certs
  • name: kibana-certificates
    secretName: kibana-certificates
    path: /usr/share/kibana/config/certs/kibana

image: "docker.elastic.co/kibana/kibana"
imageTag: "7.1.1"
imagePullPolicy: "IfNotPresent"

resources:
requests:
cpu: "100m"
memory: "500m"
limits:
cpu: "1000m"
memory: "1Gi"

protocol: https

serverHost: "0.0.0.0"

healthCheckPath: "/app/kibana"

kibanaConfig:
kibana.yml: |
xpack.security.enabled: true
server.ssl:
enabled: true
key: /usr/share/kibana/config/certs/kibana/tls.key
certificate: /usr/share/kibana/config/certs/kibana/tls.crt
xpack.security.encryptionKey: ajskfiejknckskspwu5r930128fgjvklnkdpajfe
logging.verbose: true
elasticsearch.ssl:
certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
verificationMode: certificate

podSecurityContext:
fsGroup: 1000

securityContext:
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000

serviceAccount: ""

priorityClassName: ""

antiAffinityTopologyKey: "kubernetes.io/hostname"

antiAffinity: "hard"

httpPort: 5601

maxUnavailable: 1

updateStrategy:
type: "Recreate"

service:
type: ClusterIP
port: 5601
nodePort:
annotations: {}

readinessProbe:
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 3
timeoutSeconds: 5

imagePullSecrets:
nodeSelector: {}
tolerations:
affinity: {}

nameOverride: ""
fullnameOverride: ""

and created all the p12 and pem files inside kubernetes.

However kibana cannot login successfully:
{"type":"error","@timestamp":"2019-07-12T14:12:03Z","tags":["debug","security","auth","session"],"pid":1,"level":"error","error":{"message":"Unauthorized","name":"Error","stack":"Error: Unauthorized\n at validate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:153:49)\n at Object.authenticate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:226:26)\n at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)\n at module.exports.internals.Auth.test (/usr/share/kibana/node_modules/hapi/lib/auth.js:92:54)\n at Session.get (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/session.js:56:47)\n at Authenticator.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:132:49)\n at Object.server.expose.request [as authenticate] (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:288:60)\n at Object.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/auth_redirect.js:28:60)\n at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)\n at module.exports.internals.Auth._authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:238:58)\n at authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:214:21)\n at Request._lifecycle (/usr/share/kibana/node_modules/hapi/lib/request.js:263:62)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"message":"Unauthorized"}
{"type":"log","@timestamp":"2019-07-12T14:12:03Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate user request to /app/kibana."}
{"type":"log","@timestamp":"2019-07-12T14:12:03Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate via login attempt."}
{"type":"log","@timestamp":"2019-07-12T14:12:03Z","tags":["debug","security","basic"],"pid":1,"message":"Username and password not found in payload."}
{"type":"log","@timestamp":"2019-07-12T14:12:03Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate via header."}
{"type":"log","@timestamp":"2019-07-12T14:12:03Z","tags":["debug","security","basic"],"pid":1,"message":"Authorization header is not presented."}

I can successfully curl elasticsearch:
[root@elasticsearch-master-0 elasticsearch]# curl -k -u elastic:(pass) https://elasticsearch-master:9200
{
"name" : "elasticsearch-master-0",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "Oh9jTS4qTni__GG6KHCo3w",
"version" : {
"number" : "7.1.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "7a013de",
"build_date" : "2019-05-23T14:04:00.380842Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

I dont really know what I can do

@techcraft I think you don't have kibana correctly configured to run ssl. Please look at the example at https://github.com/elastic/helm-charts/blob/master/kibana/examples/security/security.yml#L17

Cheers

Hi if you look at my kibana.yml is identical to the example in the helm security example you sent. The identation are wrong for the copy&paste. Any ideas?

{"type":"log","@timestamp":"2019-07-16T14:07:43Z","tags":["status","plugin:spaces@7.1.1","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState":"red","prevMsg":"[security_exception] failed to authenticate user [elastic], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } }"}
{"type":"log","@timestamp":"2019-07-16T14:07:44Z","tags":["debug","legacy-proxy"],"pid":1,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2019-07-16T14:07:44Z","tags":["debug","legacy-service"],"pid":1,"message":"Request will be handled by proxy GET:/app/kibana."}
{"type":"error","@timestamp":"2019-07-16T14:07:44Z","tags":["debug","security","auth","session"],"pid":1,"level":"error","error":{"message":"Unauthorized","name":"Error","stack":"Error: Unauthorized\n at validate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:153:49)\n at Object.authenticate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:226:26)\n at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)\n at module.exports.internals.Auth.test (/usr/share/kibana/node_modules/hapi/lib/auth.js:92:54)\n at Session.get (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/session.js:56:47)\n at Authenticator.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:132:49)\n at Object.server.expose.request [as authenticate] (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:288:60)\n at Object.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/auth_redirect.js:28:60)\n at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)\n at module.exports.internals.Auth._authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:238:58)\n at authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:214:21)\n at Request._lifecycle (/usr/share/kibana/node_modules/hapi/lib/request.js:263:62)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"message":"Unauthorized"}
{"type":"log","@timestamp":"2019-07-16T14:07:44Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate user request to /app/kibana."}
{"type":"log","@timestamp":"2019-07-16T14:07:44Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate via login attempt."}
{"type":"log","@timestamp":"2019-07-16T14:07:44Z","tags":["debug","security","basic"],"pid":1,"message":"Username and password not found in payload."}
{"type":"log","@timestamp":"2019-07-16T14:07:44Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate via header."}
{"type":"log","@timestamp":"2019-07-16T14:07:44Z","tags":["debug","security","basic"],"pid":1,"message":"Authorization header is not presented."}
{"type":"response","@timestamp":"2019-07-16T14:07:44Z","tags":,"pid":1,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"/"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":302,"responseTime":90,"contentLength":9},"message":"GET /app/kibana 302 90ms - 9.0B"}
{"type":"log","@timestamp":"2019-07-16T14:07:45Z","tags":["plugin","debug"],"pid":1,"message":"Checking Elasticsearch version"}

@techcraft can you please remove elasticsearchURL: "" # "http://elasticsearch-master:9200" from your configuration at all and just config it with elasticsearchHosts? (assuming that you are running the last kibana and elasticsearch versions)

Same error:
{"type":"error","@timestamp":"2019-07-16T15:59:57Z","tags":["debug","security","auth","session"],"pid":1,"level":"error","error":{"message":"Unauthorized","name":"Error","stack":"Error: Unauthorized\n at validate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:153:49)\n at Object.authenticate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:226:26)\n at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)\n at module.exports.internals.Auth.test (/usr/share/kibana/node_modules/hapi/lib/auth.js:92:54)\n at Session.get (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/session.js:56:47)\n at Authenticator.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:132:49)\n at Object.server.expose.request [as authenticate] (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:288:60)\n at Object.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/auth_redirect.js:28:60)\n at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)\n at module.exports.internals.Auth._authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:238:58)\n at authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:214:21)\n at Request._lifecycle (/usr/share/kibana/node_modules/hapi/lib/request.js:263:62)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"message":"Unauthorized"}
{"type":"log","@timestamp":"2019-07-16T15:59:57Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate user request to /app/kibana."}
{"type":"log","@timestamp":"2019-07-16T15:59:57Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate via login attempt."}
{"type":"log","@timestamp":"2019-07-16T15:59:57Z","tags":["debug","security","basic"],"pid":1,"message":"Username and password not found in payload."}
{"type":"log","@timestamp":"2019-07-16T15:59:57Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate via header."}
{"type":"log","@timestamp":"2019-07-16T15:59:57Z","tags":["debug","security","basic"],"pid":1,"message":"Authorization header is not presented."}

I commented the entry in the values file of kibana as suggested:

#elasticsearchURL: "" # "http://elasticsearch-master:9200"
elasticsearchHosts: "https://elasticsearch-master:9200"

The elasticsearch node is up&running and I can list the indexes inside with authentication:

[root@elasticsearch-master-0 elasticsearch]# curl -k -u elastic:[password] https://localhost:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana_1 vgZLE6VMQdSv9EIz3RmdVA 1 0 2 0 10.5kb 10.5kb
green open .kibana_task_manager I0-36A-mRWOwUob0joWIKQ 1 0 2 4 44.9kb 44.9kb
green open .security-7 ioi3iwf9Sce0QtXXS-SaJA 1 0 4 7 36.2kb 36.2kb

Humm , let's try the debug steps I'm also suggesting in another thread.

For a matter of testing it, can you please do the following:

  • change in your kibana security config yml the protocol to http
  • comment the configurations for the elasticsearch ssl settings
  • set logging.verbose: true in the kibana configuration
  • try to login in an anonymous tab

Cheers

Hello,
Im receiving the same error: disabled all ssl configurations and I can curl elasticsearch without certifcates:

[root@elasticsearch-master-0 elasticsearch]# curl -u elastic:citsale1 http://localhost:9200
{
  "name" : "elasticsearch-master-0",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "ayNrgP57QdGWD0VJih3yqw",
  "version" : {
    "number" : "7.1.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "7a013de",
    "build_date" : "2019-05-23T14:04:00.380842Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

When I start kibana I see the same log:

{"type":"error","@timestamp":"2019-07-17T14:10:04Z","tags":["debug","security","auth","session"],"pid":1,"level":"error","error":{"message":"Unauthorized","name":"Error","stack":"Error: Unauthorized\n    at validate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:153:49)\n    at Object.authenticate (/usr/share/kibana/node_modules/hapi-auth-cookie/lib/index.js:226:26)\n    at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)\n    at module.exports.internals.Auth.test (/usr/share/kibana/node_modules/hapi/lib/auth.js:92:54)\n    at Session.get (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/session.js:56:47)\n    at Authenticator.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:132:49)\n    at Object.server.expose.request [as authenticate] (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/authentication/authenticator.js:288:60)\n    at Object.authenticate (/usr/share/kibana/node_modules/x-pack/plugins/security/server/lib/auth_redirect.js:28:60)\n    at module.exports.internals.Manager.execute (/usr/share/kibana/node_modules/hapi/lib/toolkit.js:35:106)\n    at module.exports.internals.Auth._authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:238:58)\n    at authenticate (/usr/share/kibana/node_modules/hapi/lib/auth.js:214:21)\n    at Request._lifecycle (/usr/share/kibana/node_modules/hapi/lib/request.js:263:62)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"message":"Unauthorized"}
{"type":"log","@timestamp":"2019-07-17T14:10:04Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate user request to /app/kibana."}
{"type":"log","@timestamp":"2019-07-17T14:10:04Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate via login attempt."}
{"type":"log","@timestamp":"2019-07-17T14:10:04Z","tags":["debug","security","basic"],"pid":1,"message":"Username and password not found in payload."}
{"type":"log","@timestamp":"2019-07-17T14:10:04Z","tags":["debug","security","basic"],"pid":1,"message":"Trying to authenticate via header."}
{"type":"log","@timestamp":"2019-07-17T14:10:04Z","tags":["debug","security","basic"],"pid":1,"message":"Authorization header is not presented."}
{"type":"response","@timestamp":"2019-07-17T14:10:04Z","tags":[],"pid":1,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"user-agent":"curl/7.29.0","host":"localhost:5601","accept":"*/*"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1"},"res":{"statusCode":302,"responseTime":17,"contentLength":9},"message":"GET /app/kibana 302 17ms - 9.0B"}

I can access to the web interface but I cannot login with elastic user (Oops! Error. Try again.)

these are the actual values files:

elasticsearch.yml

clusterName: "elasticsearch"
nodeGroup: "master"

masterService: ""

roles:
  master: "true"
  ingest: "true"
  data: "true"

replicas: 1
minimumMasterNodes: 1

esMajorVersion: 7

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    # xpack.security.transport.ssl.enabled: true
    # xpack.security.transport.ssl.verification_mode: certificate
    # xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificate.p12
    # xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificate.p12
    # xpack.security.http.ssl.enabled: true
    # xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificate.p12
    # xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificate.p12

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
  - name: ELASTIC_USERNAME
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username

secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

image: "docker.elastic.co/elasticsearch/elasticsearch"
imageTag: "7.1.1"
imagePullPolicy: "IfNotPresent"

podAnnotations: {}
  # iam.amazonaws.com/role: es-cluster

esJavaOpts: "-Xmx3g -Xms3g"

resources:
  requests:
    cpu: "100m"
    memory: "3Gi"
  limits:
    cpu: "1000m"
    memory: "4Gi"

initResources: {}

networkHost: "0.0.0.0"

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 10Gi

persistence:
  enabled: true
  annotations: {}

extraVolumes: []
  # - name: extras
  #   emptyDir: {}

extraVolumeMounts: []

extraInitContainers: []


# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""


antiAffinityTopologyKey: "kubernetes.io/hostname"

antiAffinity: "hard"

# This is the node affinity settings as defined in
# https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
nodeAffinity: {}

# The default is to deploy all pods serially. By setting this to parallel all pods are started at
# the same time when bootstrapping the cluster
podManagementPolicy: "Parallel"

protocol: http
httpPort: 9200
transportPort: 9300

service:
  type: ClusterIP
  nodePort:
  annotations: {}

updateStrategy: RollingUpdate

maxUnavailable: 1

 # GroupID for the elasticsearch user. The official elastic docker images always have the id of 1000
fsGroup: 1000

# How long to wait for elasticsearch to stop gracefully
terminationGracePeriod: 120

sysctlVmMaxMapCount: 262144

readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5

# https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html#request-params wait_for_status
clusterHealthCheckParams: "wait_for_status=green&timeout=1s"

schedulerName: ""

imagePullSecrets: []
nodeSelector: {}
tolerations: []

# Enabling this will publically expose your Elasticsearch instance.
# Only enable this if you have security enabled on your cluster
ingress:
  enabled: true
  annotations: 
    kubernetes.io/ingress.class: nginx-internal
    # kubernetes.io/tls-acme: "true"
  path: /api
  hosts:
    - elastic.dominiog.lan
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

nameOverride: ""
fullnameOverride: ""

kibana.yml


elasticsearchURL: "" # "http://elasticsearch-master:9200"
elasticsearchHosts: "http://elasticsearch-master:9200"


replicas: 1

extraEnvs:
  - name: 'ELASTICSEARCH_USERNAME'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: username
  - name: 'ELASTICSEARCH_PASSWORD'
    valueFrom:
      secretKeyRef:
        name: elastic-credentials
        key: password
#  - name: MY_ENVIRONMENT_VAR
#    value: the_value_goes_here

secretMounts:
  - name: elastic-certificate-pem
    secretName: elastic-certificate-pem
    path: /usr/share/kibana/config/certs
  - name: kibana-certificates
    secretName: kibana-certificates
    path: /usr/share/kibana/config/certs/kibana

image: "docker.elastic.co/kibana/kibana"
imageTag: "7.1.1"
imagePullPolicy: "IfNotPresent"

resources:
  requests:
    cpu: "100m"
    memory: "500m"
  limits:
    cpu: "1000m"
    memory: "1Gi"

protocol: http

serverHost: "0.0.0.0"

healthCheckPath: "/app/kibana"

# Allows you to add any config files in /usr/share/kibana/config/
# such as kibana.yml
kibanaConfig:
  kibana.yml: |
    logging.verbose: true
  #   server.ssl:
  #     enabled: true
  #     key: /usr/share/kibana/config/certs/kibana/tls.key
  #     certificate: /usr/share/kibana/config/certs/kibana/tls.crt
  #   xpack.security.encryptionKey: ajskfiejknckskspwu5r930128fgjvklnkdpajfe
  #   elasticsearch.ssl: 
  #     certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
  #     verificationMode: certificate
  #   server.host: "0.0.0.0"
  #   server.port: 5601
  #   elasticsearch.hosts: "http://elasticsearch-master:9200"
  #   elasticsearch.ssl.verificationMode: none
#    xpack.monitoring.elasticsearch.username: "elastic"
#    xpack.monitoring.elasticsearch.password: "citsale1"
    
      #nestedkey: value

# If Pod Security Policy in use it may be required to specify security context as well as service account

podSecurityContext:
  fsGroup: 1000

securityContext:
  capabilities:
    drop:
    - ALL
  # readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000

serviceAccount: ""

# This is the PriorityClass settings as defined in
# https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""

antiAffinityTopologyKey: "kubernetes.io/hostname"

antiAffinity: "hard"

httpPort: 5601

# This is the max unavailable setting for the pod disruption budget
# The default value of 1 will make sure that kubernetes won't allow more than 1
# of your pods to be unavailable during maintenance
maxUnavailable: 1

updateStrategy:
  type: "Recreate"

service:
  type: ClusterIP
  port: 5601
  nodePort:
  annotations: {}


ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx-internal
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - elastic.dominiog.lan
  tls:
    - secretName: sites-crt
      hosts:
        - elastic.dominiog.lan


readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5

imagePullSecrets: []
nodeSelector: {}
tolerations: []
affinity: {}

nameOverride: ""
fullnameOverride: ""

any ideas? it seems kibana is not able to use xpack

Hi is there anyone who has deployed elasticsearch+kibana with helm in microservices using xpack?

Could you post the output of helm get elasticsearch and helm get kibana (or replace them with your actual release names). When you do, could you also make sure to put everything in code blocks.

```
output goes in here
```

To make sure that it is readable.

Some other users were having issues getting a security cluster bootstrapped. I very recently updated the security docs to make it much easier. Can you try again with the latest documentation please? https://github.com/elastic/helm-charts/tree/master/kibana#security

Hi Michael,
the problem is the ingress section: if you create the self-signed certificates and access kibana via ingress the post authorization header is blank because it is not populated by the browser but inside kibana. Everything is fine if kibana is exposed as NodePort (and all ssl/tls layer activated). The problem is how to use an ingress rule with kibana.

This doesn't sound like an issue specific to Kibana or the helm-charts. The charts allow you to configure the service and ingress rules however you like. The problem you are having is going to be the same for any service in Kubernetes that uses self signed certificates that you want behind an ingress controller. I can try to help you with this but you are going to get much better advice on a Kubernetes focused forum or in their slack group.

The problem is how to use an ingress rule with kibana.

Which ingress controller are you using?

If you take a look at the kibana chart you can see I am using it:

ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx-internal
# kubernetes.io/tls-acme: "true"
path: /
hosts:
- elastic.dominiog.lan
tls:
- secretName: sites-crt
hosts:
- elastic.dominiog.lan

but it cannot run with kibana application because the ssl self signed certification is passed inside a kibana process to localhost with a POST header. If we put an ingress (in my case the classical nginx) the browser tries to send the POST header but kibana dont accept it because it's self-signed and not accepted without the CA part.

You can see this behaviour in kibana log:
["debug","security","basic"],"pid":1,"message":"Authorization header is not presented."}

Usually applications "speak" directly with the browser, instead in this case the authentication is managed internally kibana.
I think the only way to bypass this problem is using a cert-manager module inside k8s in order to have every transiction between ingress and kibana certified by CA and so accepted.

All the configurations inside the github project use a NodePort configuration but it cannot go in a production environment

Usually applications "speak" directly with the browser, instead in this case the authentication is managed internally kibana.

It sounds like something is misconfigured with your nginx ingress controller. Internally we are using nginx ingress with Kibana and authentication enabled without issue. Can you post the version and configuration for your nginx ingress controller?

The nginx ingress annotations you have shown don't look right either. If you are running any Kubernetes application with a self signed certificate there are a few options for how to properly expose it.

  1. Set the backend protocol to "HTTPS" Annotations - Ingress-Nginx Controller and install the certificate into the nginx ingress controller.
  2. Use SSL passthrough TLS/HTTPS - Ingress-Nginx Controller
  3. Have Kibana listen on http and let nginx handle TLS (note this means you need to trust all traffic on your Kubernetes cluster and anything sharing the same internal network).

Hi Michael,
Im following your post and I have disabled ssl and used only tls.

In this way I can authenticate successfully on kibana but not using ingress but with a port-forward.

If I try to use the ingress I get a

2019/08/16 09:34:36 [error] 4063#4063: *287347 connect() failed (111: Connection refused) while connecting to upstream, client: 10.165.26.122, server: elastic.dominiog.lan, request: "GET / HTTP/1.1", upstream: "http://10.42.0.127:5601/", host: "elastic.dominiog.lan"

My ingress is:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: elastic
  annotations:
    kubernetes.io/ingress.class: nginx-internal
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/configuration-snippet: |
      rewrite ^/logs/(.*)$ /$1 break;

spec:
  rules:
  - host: elastic.dominiog.lan
    http:
      paths:
      - path: /
        backend:
          serviceName: kibana-kibana
          servicePort: 5601

Hey!

Could you post the helm get release output I asked for in my earlier comment. It's a bit hard to know what is going wrong if I can't see your full configuration.

If you try to directly connect with curl http://10.42.0.127:5601/ (run it from inside a pod that is in the cluster) is that working?

Hi Michael,
sorry I lost your request.

Elasticsearch: https://gist.github.com/Tekcraft/d13801a212c2c056c9439a923af6ff89
kibana: https://gist.github.com/Tekcraft/1e6ac1cf828cd8b2203839a060ae4708
In Kibana.yml I tried to disable the BASE PATH rewrite thinking the ingress were not able to follow the url redirection of kibana

If I try to curl inside the kibana pod:

bash-4.2$ curl http://10.42.0.127:5601 
curl: (7) Failed connect to 10.42.0.127:5601; Connection refused
bash-4.2$ curl http://localhost:5601
<empty return>

As written before, if I port forward the port I can connect kibana successfully

kubectl -n elastic port-forward kibana-kibana-8497b9cccd-5p652 5601