Kibana 7.7 using helm chart, pod exits on error

Here are the logs from the pod startup:

Tried adding values into kibana.yaml and it doesn't looked to have helped.

{"type":"log","@timestamp":"2020-05-19T22:17:26Z","tags":["debug","plugins-service"],"pid":8,"message":"Discovered 76 plugins."}
{"type":"log","@timestamp":"2020-05-19T22:17:26Z","tags":["debug","config"],"pid":8,"message":"Marking config path as handled: path"}
{"type":"log","@timestamp":"2020-05-19T22:17:26Z","tags":["debug","root"],"pid":8,"message":"shutting root down"}
{"type":"log","@timestamp":"2020-05-19T22:17:26Z","tags":["fatal","root"],"pid":8,"message":"{ ValidationError: child \"server\" fails because [child \"port\" fails because [\"port\" must be a number]]\n    at Object.exports.process (/usr/share/kibana/node_modules/joi/lib/errors.js:196:19)\n    at internals.Object._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:675:31)\n    at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:146:23)\n    at Config._commit (/usr/share/kibana/src/legacy/server/config/config.js:132:34)\n    at Config.set (/usr/share/kibana/src/legacy/server/config/config.js:103:10)\n    at Config.extendSchema (/usr/share/kibana/src/legacy/server/config/config.js:75:10)\n    at _lodash.default.each.child (/usr/share/kibana/src/legacy/server/config/config.js:64:14)\n    at arrayEach (/usr/share/kibana/node_modules/lodash/index.js:1315:13)\n    at Function.<anonymous> (/usr/share/kibana/node_modules/lodash/index.js:3393:13)\n    at Config.extendSchema (/usr/share/kibana/src/legacy/server/config/config.js:63:30)\n    at new Config (/usr/share/kibana/src/legacy/server/config/config.js:54:10)\n    at Function.withDefaultSchema (/usr/share/kibana/src/legacy/server/config/config.js:48:12)\n    at defaultConfig (/usr/share/kibana/src/legacy/plugin_discovery/find_plugin_specs.js:44:25)\n    at findLegacyPluginSpecs (/usr/share/kibana/src/core/server/legacy/plugins/find_legacy_plugin_specs.js:38:63)\n    at LegacyService.discoverPlugins (/usr/share/kibana/src/core/server/legacy/legacy_service.js:98:50) name: 'ValidationError' }"}
{"type":"log","@timestamp":"2020-05-19T22:17:26Z","tags":["debug","server"],"pid":8,"message":"stopping server"}
{"type":"log","@timestamp":"2020-05-19T22:17:26Z","tags":["debug","legacy-service"],"pid":8,"message":"stopping legacy service"}
{"type":"log","@timestamp":"2020-05-19T22:17:26Z","tags":["debug","plugins-service"],"pid":8,"message":"Stopping plugins service"}
{"type":"log","@timestamp":"2020-05-19T22:17:26Z","tags":["debug","elasticsearch-service"],"pid":8,"message":"Stopping elasticsearch service"}

 FATAL  ValidationError: child "server" fails because [child "port" fails because ["port" must be a number]]

Here is the deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: kibana
  name: kibana-kibana
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kibana

  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - env:
        - name: ELASTICSEARCH_URL
          value: https://es7-default:9200
        - name: SERVER_HOST
          value: 0.0.0.0
        - name: ELASTICSEARCH_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: elastic-credentials
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: elastic-credentials
        - name: KIBANA_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              key: encryptionkey
              name: kibana
        image: docker.elastic.co/kibana/kibana:7.7.0
        imagePullPolicy: IfNotPresent
        name: kibana
        ports:
        - containerPort: 5601
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              http () {
                  local path="${1}"
                  set -- -XGET -s --fail -L

                  if [ -n "${ELASTICSEARCH_USERNAME}" ] && [ -n "${ELASTICSEARCH_PASSWORD}" ]; then
                    set -- "$@" -u "${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD}"
                  fi

                  STATUS=$(curl --output /dev/null --write-out "%{http_code}" -k "$@" "https://localhost:5601${path}")
                  if [[ "${STATUS}" -eq 200 ]]; then
                    exit 0
                  fi

                  echo "Error: Got HTTP code ${STATUS} but expected a 200"
                  exit 1
              }

              http "/app/kibana"
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "1"
            memory: 2Gi
          requests:
            cpu: "1"
            memory: 2Gi
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/kibana/config/certs
          name: elastic-certificate-pem
        - mountPath: /usr/share/kibana/config/kibana.yml
          name: kibanaconfig
          subPath: kibana.yml
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
      terminationGracePeriodSeconds: 30
      volumes:
      - name: elastic-certificate-pem
        secret:
          defaultMode: 420
          secretName: elastic-certificate-pem
      - configMap:
          defaultMode: 420
          name: kibana-es7-default-kibana-config
        name: kibanaconfig

Here is the configmap:

apiVersion: v1
data:
  kibana.yml: |
    elasticsearch.hosts: https://es7-default:9200
    elasticsearch.url: https://es7-default:9200
    elasticsearch.ssl.verificationMode: none
    logging.verbose: true
    server.port: 5601

    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/elastic-certificate.pem
      certificate: /usr/share/kibana/config/certs/elastic-certificate.pem
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate

we discovered that we had a previous service some time ago defined as simply "SERVER" and kube was injecting an env var for "SERVER_PORT" of SERVER_PORT= tcp://10.0.20.191:80 ! Which kibana with full integrity honors using env var over config file and hence our error; Adding SERVER_PORT=5601 to extraVars fixed it.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.