Custom kibana settings in ECK

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

It appears that you have to do manual configuration of ES if you want to change items in the kibana.yml such as elasticsearch.requestHeadersWhitelist? is this correct and we cant use elasticsearchRef to do all the username and cert stuff for us?

2

Yes, you can combine both. We merge the settings you specify in the config section with the settings we derive from the elasticsearchRef

apiVersion: kibana.k8s.elastic.co/v1alpha1
kind: Kibana
metadata:
  name: kibana-sample
spec:
  version: 7.4.0
  nodeCount: 1
  config:
     elasticsearch.requestHeadersWhitelist:
     - authorization # or whatever you want here
  elasticsearchRef:
    name: "elasticsearch-sample"

In this example ECK takes care of the username, password, Elasticsearch URL and certificate authority setup but you can set a custom request header whitelist.

3 Likes
3

missed that it was config: and not elasticsearch:

Thanks!!

4

Hi,

I'm having a similar problem. I followed the guideline and created some custom configurations, but unfortunately they are not merged.

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: default
  labels:
    k8s-app: kibana
spec:
  version: 7.5.2
  count: 1
  config:
    xpack.security.enabled:
    - true
    xpack.security.audit.enabled:
    - true
    elasticsearch.username:
    - elastic
    elasticsearch.ssl.certificateAuthorities:
    - /etc/certs/ca.crt
    xpack.monitoring.kibana.collection.enabled:
    - false
    xpack.monitoring.ui.container.elasticsearch.enabled: false
  elasticsearchRef:
    name: "***"
...

When I have a look at the created config via kubectl I get different values:

[user@host]$ kubectl exec -it kibana -- cat config/kibana.yml
elasticsearch:
  hosts:
  - https://***:9200
  password: ***
  ssl:
    certificateAuthorities: /etc/certs/ca.crt
    verificationMode: certificate
  username: elastic
server:
  host: "0"
  name: kibana
  ssl:
    certificate: /mnt/elastic-internal/http-certs/tls.crt
    enabled: true
    key: /mnt/elastic-internal/http-certs/tls.key
xpack:
  monitoring:
    ui:
      container:
        elasticsearch:
          enabled: true
  security:
    audit:
      enabled: true
    enabled: true

Is this a known problem or did I do something wrong in my configuration file?

Greetings,

Samuel

5

Probably unrelated but these settings values should not be arrays, but single values. For example:

xpack.security.audit.enabled: true

As to why the values are not propagated correctly into the pod:

First, just to double check are we looking at the correct pod?

because I would expect the pod to be named something like kibana-kb-SOME_HASH

If it is the correct pod, we could double check if the reconciliation actually went through and updated the configuration. For that it would be worth checking the operator logs as described here:

6

Hi Peter,

Thanks for getting back to me. Yes the podname is corrent (I sanatized the -kb-SOME_HASH) to make it more readable. I also tried setting the values to single values as you suggested but there is no change in the behavior.

  config:
    xpack.security.enabled: true
    xpack.security.audit.enabled: true
    elasticsearch.username: elastic
    elasticsearch.ssl.certificateAuthorities:
    - /etc/certs/ca.crt
    xpack.monitoring.kibana.collection.enabled: false
    xpack.monitoring.ui.container.elasticsearch.enabled: false

Below you can find the operator logs (only errors) which show up when recreating the kibana instance. After this the error messages just keep repeating themselves. There might be a rather obvious reason("User-configured use is unsupported") , why I can not set the parameters. If this is the case, how am I supposed to set other values?

{"level":"error","@timestamp":"2020-02-05T13:34:26.968Z","logger":"controller runtime.controller","message":"Reconciler error","ver":"1.0.0-6881438d","controller":"kibana association-controller","request":"default/secmon-kibana","error":"Operation cannot be fulfilled on kibanas.kibana.k8s.elastic.co \"secmon-kibana\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io apimachinery@v0.0.0-20191028221656-72ed19daf4bb/pkg/util/wait/wait.go:152\nk8s.io apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191028221656-72ed19daf4bb/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191028221656-72ed19daf4bb/pkg/util/wait/wait.go:88"}
{"level":"info","@timestamp":"2020-02-05T13:34:28.054Z","logger":"elasticsearch-controller","message":"Elasticsearch manifest has warnings. Proceed at your own risk. [spec.nodeSets[0].config.xpack.security.enabled: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported, spec.nodeSets[0].config.xpack.security.http.ssl.enabled: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported, spec.nodeSets[0].config.xpack.security.transport.ssl.enabled: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported, spec.nodeSets[0].config.xpack.security.transport.ssl.verification_mode: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported, spec.nodeSets[1].config.xpack.security.enabled: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported, spec.nodeSets[1].config.xpack.security.http.ssl.enabled: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported, spec.nodeSets[1].config.xpack.security.transport.ssl.enabled: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported, spec.nodeSets[1].config.xpack.security.transport.ssl.verification_mode: Forbidden: Configuration setting is reserved for internal use. User-configured use is unsupported]","ver":"1.0.0-6881438d","namespace":"default","es_name":"secmon"}
{"level":"error","@timestamp":"2020-02-05T13:34:29.999Z","logger":"controller-runtime.controller","message":"Reconciler error","ver":"1.0.0-6881438d","controller":"kibana-controller","request":"default/secmon-kibana","error":"Deployment.apps \"secmon-kibana-kb\" is invalid: spec.template.spec.containers[0].image: Required value","errorCauses":[{"error":"Deployment.apps \"secmon-kibana-kb\" is invalid: spec.template.spec.containers[0].image: Required value"}],"stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191028221656-72ed19daf4bb/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191028221656-72ed19daf4bb/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191028221656-72ed19daf4bb/pkg/util/wait/wait.go:88"}

Greetings

Samuel

7

These are just warnings and are referring to your Elasticsearch configuration, where you are apparently overriding values that we consider managed by ECK itself (see Settings managed by ECK | Elastic Cloud on Kubernetes [2.10] | Elastic)

If you want to set your own settings in the config section you can just do that, no need to include all the settings internally managed by ECK. ECK will merge your settings with the settings it generates internally, where your settings take precedence over anything set by ECK.

As to why you Kibana configuration does not line up with what you have in your manifest:

This seems relevant as it indicates that ECK was not able to update your Kibana deployment. So you are still seeing an old version of the deployment. Are you by any chance setting an empty Docker image or overriding the podTemplate for the Kibana deployment, in any way?

8

That might have been the problem. I destroyed every manually and now it seems to work fine. Thanks for helping out.