License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] missing authentication credentials for REST request [/_xpack]

[KatiukOleg (Oleg)]

Hi guys, I have some trouble with Elastic and Kibana. I launched them in Azure through Azure Kubernetes Service and was using standard (not oss) images for it. So, Elasticsearch is up and running. I configured passwords for built-in user via elasticsearch-setup-passwords and can access to Elastic from Kibana container with kibana user credentials. But when I'm trying access to Kibana via browser I get - Kibana server is not ready yet. I was looking to logs and find this warning message

{"type":"log","@timestamp":"2019-06-05T13:37:42Z","tags":["license","warning","xpack"],"pid":1,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_exception] missing authentication credentials for REST request [/_xpack], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"missing authentication credentials for REST request [/_xpack]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}"}

My elasticdeployment yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-elasticsearch-prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-elasticsearch-prod
  template:
    metadata:
      labels:
        app: azure-elasticsearch-prod
    spec:
      containers:
      - name: azure-elasticsearch-prod
        image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1
        resources:
          limits:
            memory: 1Gi
            cpu: 1000m
          requests:
            memory: 512Mi
            cpu: 350m
        ports:
        - containerPort: 9200
        - containerPort: 9300
        env:
        - name: bootstrap.memory_lock
          value: "true"
        - name: xpack.security.enabled
          value: "true"
        - name: discovery.type
          value: single-node
        - name: ES_JAVA_OPTS
          value: "-Xms512m -Xmx512m"
        - name: ES_HEAP_SIZE
          value: 2Gi

Mu Kibana yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-kibana-prod
spec:
  replicas: 1
  selector:
    matchLabels:
      app: azure-kibana-prod
  template:
    metadata:
      labels:
        app: azure-kibana-prod
    spec:
      containers:
      - name: azure-kibana-prod
        image: docker.elastic.co/kibana/kibana:7.1.1
        resources:
          requests:
            cpu: 100m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi
        ports:
        - containerPort: 5601
        env:
        - name: ELASTICSEARCH_HOSTS        
          value: http://azure-elasticsearch-prod:9200
        - name: xpack.security.enabled
          value: "true"
        - name: xpack.monitoring.enabled
          value: "true"
        - name: xpack.reporting.enabled
          value: "true"
        - name: xpack.reporting.encryptionKey
          value: "myKey"
        - name: elasticsearch.username
          value: kibana
        - name: elasticsearch.password
          value: myPassword
        - name: xpack.monitoring.elasticsearch.username
          value: kibana
        - name: xpack.monitoring.elasticsearch.password
          value: myPassword

I'll be thankful for any response

[ikakavas (Ioannis Kakavas)]

Hi there,

Please use </> or (```) to format your post ( the config snippets ). YAML is very sensitive to syntax and it's really hard to see if there is an error or something missing there without the structure being obvious. You can use the preview panel on the right while you edit your answer to see how your post will look like

[KatiukOleg (Oleg)]

Hi Ioannis, thanks for this formatting spot.

[ikakavas (Ioannis Kakavas)]

 env:
        - name: ELASTICSEARCH_HOSTS        
          value: http://azure-elasticsearch-prod:9200
        - name: xpack.security.enabled
          value: "true"
        - name: xpack.monitoring.enabled
          value: "true"
        - name: xpack.reporting.enabled
          value: "true"
        - name: xpack.reporting.encryptionKey
          value: "myKey"
        - name: elasticsearch.username
          value: kibana
        - name: elasticsearch.password
          value: myPassword
        - name: xpack.monitoring.elasticsearch.username
          value: kibana
        - name: xpack.monitoring.elasticsearch.password
          value: myPassword

Looking at https://www.elastic.co/guide/en/kibana/current/docker.html#environment-variable-config , it would seem that you need to pass all configuration parameters in the format of ELASTICSEARCH_HOSTS and not i.e. xpack.security.enabled.

This malformed configuration is consistent with what you see as behavior, that is Kibana trying to connect to elasticsearch without a username and password ( as it is not passed elasticsearch.password and elasticsearch.username that should be ELASTICSEARCH_PASSWORD and ELASTICSEARCH_USERNAME instead )

[KatiukOleg (Oleg)]

Ioannis, it resolved my problem. Using upper case format help me. Thanks a lot