Beats deployment error on ECK 1.2.0

ssurenr · July 22, 2020, 9:49pm

I tried to deploy a simple metrics beat CRD with the following manifest (Taken from the samples).

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: heartbeat
  namespace: elastic-system
spec:
  type: heartbeat
  version: 7.8.0
  elasticsearchRef:
    name: observe
    namespace: elastic-system
  kibanaRef:
    name: observe
    namespace: elastic-system
  config:
    heartbeat.monitors:
      - type: tcp
        schedule: '@every 5s'
        hosts: ["observe-es-http.elastic-system.svc:9200"]
      - type: tcp
        schedule: '@every 5s'
        hosts: ["observe-kb-http.elastic-system.svc:5601"]
  deployment:
    replicas: 1
    podTemplate:
      spec:
        securityContext:
          runAsUser: 0

After it is deployed, only the following resources are created:

NAMESPACE       NAME                             READY  REASON  AGE
elastic-system  Beat/heartbeat                   -              8m23s
elastic-system  ├─Secret/heartbeat-beat-es-ca    -              8m23s
elastic-system  ├─Secret/heartbeat-beat-kb-user  -              8m23s
elastic-system  └─Secret/heartbeat-beat-user     -              8m23s

I see the following error on elastic operator,

`{"log.level":"error","@timestamp":"2020-07-22T21:37:41.732Z","log.logger":"controller-runtime.controller","message":"Reconciler error","service.version":"1.2.0-51ae5fd4","service.type":"eck","ecs.version":"1.4.0","controller":"beat-es-association-controller","request":"elastic-system/heartbeat","error":"Operation cannot be fulfilled on beats.beat.k8s.elastic.co \"heartbeat\": the object has been modified; please apply your changes to the latest version and try again","error.stack_trace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88"}`
`{"log.level":"error","@timestamp":"2020-07-22T21:37:42.219Z","log.logger":"controller-runtime.controller","message":"Reconciler error","service.version":"1.2.0-51ae5fd4","service.type":"eck","ecs.version":"1.4.0","controller":"beat-es-association-controller","request":"elastic-system/heartbeat","error":"Operation cannot be fulfilled on beats.beat.k8s.elastic.co \"heartbeat\": the object has been modified; please apply your changes to the latest version and try again","error.stack_trace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88"}`

My stack looks like this:

NAME                                                 HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch.elasticsearch.k8s.elastic.co/observe   green    6       7.8.0     Ready   50d

NAME                                   HEALTH   NODES   VERSION   AGE
kibana.kibana.k8s.elastic.co/observe   green    2       7.8.0     50d

NAME                                   HEALTH   NODES   VERSION   AGE
apmserver.apm.k8s.elastic.co/observe   green    2       7.8.0     6d1h

NAME                                 HEALTH   AVAILABLE   EXPECTED   TYPE        VERSION   AGE
beat.beat.k8s.elastic.co/heartbeat                                   heartbeat   7.8.0     15m

Is there any problem with the manifest or this is a bug?

dkow · July 23, 2020, 6:01am

Hi @ssurenr, thanks for reporting this, it's not expected. I am able to create a Heartbeat deployment with your exact spec without any issues, so let's try to dig in further. Few questions to help with that:

Could you provide the output of kubectl describe -n elastic-system beat heartbeat?
Could you try without kibanaRef field in your spec? I'm not sure if you are using ECK CAs, but I'd expect to also see Secret/heartbeat-beat-kibana-ca in your kubectl tree output.
Could you provide more details about the environment - k8s version, if it's managed - which provider, is PSP enabled?
Is ECK validating webhook (elastic-webhook.k8s.elastic.co) installed?
Are there any interesting events in kubectl get event --namespace elastic-system --field-selector involvedObject.name=heartbeat? You can paste the output here too.

Given that only the secrets for associated resources are created, it seems that a validation or config generation is somehow failing. The errors from the operator you see can be expected and it's ok if they only appear temporarily.

ssurenr · July 23, 2020, 10:23am

Thanks for checking on this @dkow. Here are the details.

Some of the events are cleared since yesterday. I redeployed the beats and here it is.

Name:         heartbeat
Namespace:    elastic-system
Labels:       <none>
Annotations:  association.k8s.elastic.co/es-conf:
                {"authSecretName":"heartbeat-beat-user","authSecretKey":"elastic-system-heartbeat-beat-user","caCertProvided":true,"caSecretName":"heartbe...
              association.k8s.elastic.co/kb-conf:
                {"authSecretName":"heartbeat-beat-kb-user","authSecretKey":"elastic-system-heartbeat-beat-kb-user","caCertProvided":false,"caSecretName":"...
              common.k8s.elastic.co/controller-version: 1.2.0
API Version:  beat.k8s.elastic.co/v1beta1
Kind:         Beat
Metadata:
  Creation Timestamp:  2020-07-23T09:42:27Z
  Generation:          2
  Resource Version:    81919898
  Self Link:           /apis/beat.k8s.elastic.co/v1beta1/namespaces/elastic-system/beats/heartbeat
  UID:                 2e2642e9-c334-4ee9-bfab-54e0a2d91981
Spec:
  Config:
    heartbeat.monitors:
      Hosts:
        observe-es-http.elastic-system.svc:9200
      Schedule:  @every 5s
      Type:      tcp
      Hosts:
        observe-kb-http.elastic-system.svc:5601
      Schedule:  @every 5s
      Type:      tcp
  Deployment:
    Pod Template:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
        Containers:  <nil>
        Security Context:
          Run As User:  0
    Replicas:           1
  Elasticsearch Ref:
    Name:  observe
  Kibana Ref:
    Name:   observe
  Type:     heartbeat
  Version:  7.8.0
Status:
  Elasticsearch Association Status:  Established
  Kibana Association Status:         Established
Events:
  Type     Reason                   Age                From                                Message
  ----     ------                   ----               ----                                -------
  Warning  AssociationError         29s (x7 over 29s)  beat-controller                     Association backend for elasticsearch is not configured
  Normal   AssociationStatusChange  29s                beat-kibana-association-controller  Association status changed from [] to [Established]
  Warning  AssociationError         28s (x2 over 28s)  beat-controller                     Association backend for kibana is not configured
  Normal   AssociationStatusChange  28s                beat-es-association-controller      Association status changed from [] to [Established]

Removing Kibana
This is interesting, as soon as I removed kibanaRef in the manifest, the heartbeat works as expected. By the way, All the CAs are generated by ECK.

NAMESPACE       NAME                                                 READY  REASON  AGE
elastic-system  Beat/heartbeat                                       -              2m19s
elastic-system  ├─Deployment/heartbeat-beat-heartbeat                -              17s
elastic-system  │ └─ReplicaSet/heartbeat-beat-heartbeat-55b8df5874   -              16s
elastic-system  │   └─Pod/heartbeat-beat-heartbeat-55b8df5874-5vbtx  True           16s
elastic-system  ├─Secret/heartbeat-beat-es-ca                        -              2m19s
elastic-system  ├─Secret/heartbeat-beat-heartbeat-config             -              17s
elastic-system  └─Secret/heartbeat-beat-user                         -              2m19s

Adding the kibanaRef back to the manifest seems to have no effect.

NAMESPACE       NAME                                                 READY  REASON  AGE
elastic-system  Beat/heartbeat                                       -              3m31s
elastic-system  ├─Deployment/heartbeat-beat-heartbeat                -              89s
elastic-system  │ └─ReplicaSet/heartbeat-beat-heartbeat-55b8df5874   -              88s
elastic-system  │   └─Pod/heartbeat-beat-heartbeat-55b8df5874-5vbtx  True           88s
elastic-system  ├─Secret/heartbeat-beat-es-ca                        -              3m31s
elastic-system  ├─Secret/heartbeat-beat-heartbeat-config             -              89s
elastic-system  ├─Secret/heartbeat-beat-kb-user                      -              7s
elastic-system  └─Secret/heartbeat-beat-user                         -              3m31s

This is a bare-metal kubernetes. It is a vanilla kubernetes deployment using kubespray. We are in 1.17.0 and the defaults PSPs should not cause a problem as the ES, Kibana and APM instances are run by the ECK in this cluster with no problem.

No. It is not enabled.

Nothing significant. Here are the output.

LAST SEEN   TYPE      REASON                    OBJECT           MESSAGE
38m         Warning   AssociationError          beat/heartbeat   Association backend for elasticsearch is not configured
38m         Normal    AssociationStatusChange   beat/heartbeat   Association status changed from [] to [Established]
38m         Warning   AssociationError          beat/heartbeat   Association backend for kibana is not configured
38m         Normal    AssociationStatusChange   beat/heartbeat   Association status changed from [] to [Established]
35m         Warning   AssociationError          beat/heartbeat   Association backend for elasticsearch is not configured
35m         Normal    AssociationStatusChange   beat/heartbeat   Association status changed from [] to [Established]
21m         Warning   AssociationError          beat/heartbeat   Association backend for kibana is not configured
21m         Normal    AssociationStatusChange   beat/heartbeat   Association status changed from [] to [Established]
15m         Warning   AssociationError          beat/heartbeat   Association backend for elasticsearch is not configured
11m         Normal    AssociationStatusChange   beat/heartbeat   Association status changed from [] to [Established]
11m         Warning   AssociationError          beat/heartbeat   Association backend for kibana is not configured
15m         Normal    AssociationStatusChange   beat/heartbeat   Association status changed from [] to [Established]
13m         Normal    AssociationStatusChange   beat/heartbeat   Association status changed from [Established] to []

dkow · July 24, 2020, 10:41pm

Thanks @ssurenr for your detailed response. Unblocking by kibanaRef removal lead me to finding an issue that could potentially be a root cause of the behavior you've seen.

Was TLS disabled in your Kibana? If yes, that aligns with my findings. If not, could you provide the output of kubectl get -o yaml -n elastic-system kibana observe?

ssurenr · July 27, 2020, 11:36am

Spot on. Kibana is using HTTP. Thank you @dkow. I will follow up on the bug.

dkow · July 28, 2020, 10:18am

Thanks for confirming. To follow up for anyone looking here - this issue is fixed now and will be shipped in 1.2.1 release soon. We've also added additional comments directly in the yaml files and in the release notes.

Topic		Replies	Views
ECK - quickstart - heartbeat example disrupts filebeat example Elastic Cloud on Kubernetes (ECK) docker	1	397	August 16, 2022
Use metricbeat to monitor elasticsearch/kibana/beats Elastic Cloud on Kubernetes (ECK) elastic-stack-monitoring , docker	14	1957	June 29, 2022
Metricbeats can't recognize kubernetes dns Beats docker , metricbeat	2	525	February 23, 2022
Metricbeat examples on ECK not working properly Beats docker , metricbeat	1	481	October 26, 2021
Elasticsearch Elastic Cloud on Kubernetes (ECK)	3	593	May 10, 2023

Beats deployment error on ECK 1.2.0

Related topics