# Beats deployment error on ECK 1.2.0 **URL:** https://discuss.elastic.co/t/beats-deployment-error-on-eck-1-2-0/242260 **Category:** Elastic Cloud on Kubernetes (ECK) **Created:** [July 22, 2020, 9:49pm UTC](https://discuss.elastic.co/t/beats-deployment-error-on-eck-1-2-0/242260 "2020-07-22T21:49:22Z") **Posts on this page:** 7 **Page:** 1
### Author: ![ssurenr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssurenr/32/72641_2.png) [@ssurenr](https://discuss.elastic.co/u/ssurenr) #### Post date: [July 22, 2020, 9:49pm UTC](https://discuss.elastic.co/t/beats-deployment-error-on-eck-1-2-0/242260/1 "2020-07-22T21:49:22Z")
I tried to deploy a simple metrics beat CRD with the following manifest (Taken from the samples). ``` apiVersion: beat.k8s.elastic.co/v1beta1 kind: Beat metadata: name: heartbeat namespace: elastic-system spec: type: heartbeat version: 7.8.0 elasticsearchRef: name: observe namespace: elastic-system kibanaRef: name: observe namespace: elastic-system config: heartbeat.monitors: - type: tcp schedule: '@every 5s' hosts: ["observe-es-http.elastic-system.svc:9200"] - type: tcp schedule: '@every 5s' hosts: ["observe-kb-http.elastic-system.svc:5601"] deployment: replicas: 1 podTemplate: spec: securityContext: runAsUser: 0 ``` After it is deployed, only the following resources are created: ``` NAMESPACE NAME READY REASON AGE elastic-system Beat/heartbeat - 8m23s elastic-system ├─Secret/heartbeat-beat-es-ca - 8m23s elastic-system ├─Secret/heartbeat-beat-kb-user - 8m23s elastic-system └─Secret/heartbeat-beat-user - 8m23s ``` I see the following error on elastic operator, ``` `{"log.level":"error","@timestamp":"2020-07-22T21:37:41.732Z","log.logger":"controller-runtime.controller","message":"Reconciler error","service.version":"1.2.0-51ae5fd4","service.type":"eck","ecs.version":"1.4.0","controller":"beat-es-association-controller","request":"elastic-system/heartbeat","error":"Operation cannot be fulfilled on beats.beat.k8s.elastic.co \"heartbeat\": the object has been modified; please apply your changes to the latest version and try again","error.stack_trace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88"}` `{"log.level":"error","@timestamp":"2020-07-22T21:37:42.219Z","log.logger":"controller-runtime.controller","message":"Reconciler error","service.version":"1.2.0-51ae5fd4","service.type":"eck","ecs.version":"1.4.0","controller":"beat-es-association-controller","request":"elastic-system/heartbeat","error":"Operation cannot be fulfilled on beats.beat.k8s.elastic.co \"heartbeat\": the object has been modified; please apply your changes to the latest version and try again","error.stack_trace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88"}` ``` My stack looks like this: ``` NAME HEALTH NODES VERSION PHASE AGE elasticsearch.elasticsearch.k8s.elastic.co/observe green 6 7.8.0 Ready 50d NAME HEALTH NODES VERSION AGE kibana.kibana.k8s.elastic.co/observe green 2 7.8.0 50d NAME HEALTH NODES VERSION AGE apmserver.apm.k8s.elastic.co/observe green 2 7.8.0 6d1h NAME HEALTH AVAILABLE EXPECTED TYPE VERSION AGE beat.beat.k8s.elastic.co/heartbeat heartbeat 7.8.0 15m ``` Is there any problem with the manifest or this is a bug? ---
### Author: ![dkow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkow/32/47340_2.png) [@dkow](https://discuss.elastic.co/u/dkow) #### Post date: [July 23, 2020, 6:01am UTC](https://discuss.elastic.co/t/beats-deployment-error-on-eck-1-2-0/242260/2 "2020-07-23T06:01:52Z")
Hi @ssurenr, thanks for reporting this, it's not expected. I am able to create a Heartbeat deployment with your exact spec without any issues, so let's try to dig in further. Few questions to help with that: 1. Could you provide the output of `kubectl describe -n elastic-system beat heartbeat`? 2. Could you try without `kibanaRef` field in your spec? I'm not sure if you are using ECK CAs, but I'd expect to also see `Secret/heartbeat-beat-kibana-ca` in your `kubectl tree` output. 3. Could you provide more details about the environment - k8s version, if it's managed - which provider, is PSP enabled? 4. Is ECK validating webhook (`elastic-webhook.k8s.elastic.co`) installed? 5. Are there any interesting events in `kubectl get event --namespace elastic-system --field-selector involvedObject.name=heartbeat`? You can paste the output here too. Given that only the secrets for associated resources are created, it seems that a validation or config generation is somehow failing. The errors from the operator you see can be expected and it's ok if they only appear temporarily. ---
### Author: ![ssurenr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssurenr/32/72641_2.png) [@ssurenr](https://discuss.elastic.co/u/ssurenr) #### Post date: [July 23, 2020, 10:23am UTC](https://discuss.elastic.co/t/beats-deployment-error-on-eck-1-2-0/242260/3 "2020-07-23T10:23:50Z")
Thanks for checking on this @dkow. Here are the details. > [@dkow](#): > > Could you provide the output of `kubectl describe -n elastic-system beat heartbeat` ? Some of the events are cleared since yesterday. I redeployed the beats and here it is. ``` Name: heartbeat Namespace: elastic-system Labels: Annotations: association.k8s.elastic.co/es-conf: {"authSecretName":"heartbeat-beat-user","authSecretKey":"elastic-system-heartbeat-beat-user","caCertProvided":true,"caSecretName":"heartbe... association.k8s.elastic.co/kb-conf: {"authSecretName":"heartbeat-beat-kb-user","authSecretKey":"elastic-system-heartbeat-beat-kb-user","caCertProvided":false,"caSecretName":"... common.k8s.elastic.co/controller-version: 1.2.0 API Version: beat.k8s.elastic.co/v1beta1 Kind: Beat Metadata: Creation Timestamp: 2020-07-23T09:42:27Z Generation: 2 Resource Version: 81919898 Self Link: /apis/beat.k8s.elastic.co/v1beta1/namespaces/elastic-system/beats/heartbeat UID: 2e2642e9-c334-4ee9-bfab-54e0a2d91981 Spec: Config: heartbeat.monitors: Hosts: observe-es-http.elastic-system.svc:9200 Schedule: @every 5s Type: tcp Hosts: observe-kb-http.elastic-system.svc:5601 Schedule: @every 5s Type: tcp Deployment: Pod Template: Metadata: Creation Timestamp: Spec: Containers: Security Context: Run As User: 0 Replicas: 1 Elasticsearch Ref: Name: observe Kibana Ref: Name: observe Type: heartbeat Version: 7.8.0 Status: Elasticsearch Association Status: Established Kibana Association Status: Established Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning AssociationError 29s (x7 over 29s) beat-controller Association backend for elasticsearch is not configured Normal AssociationStatusChange 29s beat-kibana-association-controller Association status changed from [] to [Established] Warning AssociationError 28s (x2 over 28s) beat-controller Association backend for kibana is not configured Normal AssociationStatusChange 28s beat-es-association-controller Association status changed from [] to [Established] ``` **Removing Kibana** This is interesting, as soon as I removed kibanaRef in the manifest, the heartbeat works as expected. By the way, All the CAs are generated by ECK. ``` NAMESPACE NAME READY REASON AGE elastic-system Beat/heartbeat - 2m19s elastic-system ├─Deployment/heartbeat-beat-heartbeat - 17s elastic-system │ └─ReplicaSet/heartbeat-beat-heartbeat-55b8df5874 - 16s elastic-system │ └─Pod/heartbeat-beat-heartbeat-55b8df5874-5vbtx True 16s elastic-system ├─Secret/heartbeat-beat-es-ca - 2m19s elastic-system ├─Secret/heartbeat-beat-heartbeat-config - 17s elastic-system └─Secret/heartbeat-beat-user - 2m19s ``` Adding the kibanaRef back to the manifest seems to have no effect. ``` NAMESPACE NAME READY REASON AGE elastic-system Beat/heartbeat - 3m31s elastic-system ├─Deployment/heartbeat-beat-heartbeat - 89s elastic-system │ └─ReplicaSet/heartbeat-beat-heartbeat-55b8df5874 - 88s elastic-system │ └─Pod/heartbeat-beat-heartbeat-55b8df5874-5vbtx True 88s elastic-system ├─Secret/heartbeat-beat-es-ca - 3m31s elastic-system ├─Secret/heartbeat-beat-heartbeat-config - 89s elastic-system ├─Secret/heartbeat-beat-kb-user - 7s elastic-system └─Secret/heartbeat-beat-user - 3m31s ``` > [@dkow](#): > > Could you provide more details about the environment - k8s version, if it's managed - which provider, is PSP enabled? This is a bare-metal kubernetes. It is a vanilla kubernetes deployment using kubespray. We are in 1.17.0 and the defaults PSPs should not cause a problem as the ES, Kibana and APM instances are run by the ECK in this cluster with no problem. > [@dkow](#): > > - Is ECK validating webhook ( `elastic-webhook.k8s.elastic.co` ) installed? No. It is not enabled. > [@dkow](#): > > Are there any interesting events in `kubectl get event --namespace elastic-system --field-selector involvedObject.name=heartbeat` ? You can paste the output here too. Nothing significant. Here are the output. ``` LAST SEEN TYPE REASON OBJECT MESSAGE 38m Warning AssociationError beat/heartbeat Association backend for elasticsearch is not configured 38m Normal AssociationStatusChange beat/heartbeat Association status changed from [] to [Established] 38m Warning AssociationError beat/heartbeat Association backend for kibana is not configured 38m Normal AssociationStatusChange beat/heartbeat Association status changed from [] to [Established] 35m Warning AssociationError beat/heartbeat Association backend for elasticsearch is not configured 35m Normal AssociationStatusChange beat/heartbeat Association status changed from [] to [Established] 21m Warning AssociationError beat/heartbeat Association backend for kibana is not configured 21m Normal AssociationStatusChange beat/heartbeat Association status changed from [] to [Established] 15m Warning AssociationError beat/heartbeat Association backend for elasticsearch is not configured 11m Normal AssociationStatusChange beat/heartbeat Association status changed from [] to [Established] 11m Warning AssociationError beat/heartbeat Association backend for kibana is not configured 15m Normal AssociationStatusChange beat/heartbeat Association status changed from [] to [Established] 13m Normal AssociationStatusChange beat/heartbeat Association status changed from [Established] to [] ``` ---
### Author: ![dkow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkow/32/47340_2.png) [@dkow](https://discuss.elastic.co/u/dkow) #### Post date: [July 24, 2020, 10:41pm UTC](https://discuss.elastic.co/t/beats-deployment-error-on-eck-1-2-0/242260/4 "2020-07-24T22:41:06Z")
Thanks @ssurenr for your detailed response. Unblocking by `kibanaRef` removal lead me to finding an [issue](https://github.com/elastic/cloud-on-k8s/issues/3523) that could potentially be a root cause of the behavior you've seen. Was TLS disabled in your Kibana? If yes, that aligns with my findings. If not, could you provide the output of `kubectl get -o yaml -n elastic-system kibana observe`? ---
### Author: ![ssurenr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ssurenr/32/72641_2.png) [@ssurenr](https://discuss.elastic.co/u/ssurenr) #### Post date: [July 27, 2020, 11:36am UTC](https://discuss.elastic.co/t/beats-deployment-error-on-eck-1-2-0/242260/5 "2020-07-27T11:36:31Z")
Spot on. Kibana is using HTTP. Thank you @dkow. I will follow up on the bug. ---
### Author: ![dkow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkow/32/47340_2.png) [@dkow](https://discuss.elastic.co/u/dkow) #### Post date: [July 28, 2020, 10:18am UTC](https://discuss.elastic.co/t/beats-deployment-error-on-eck-1-2-0/242260/6 "2020-07-28T10:18:13Z")
Thanks for confirming. To follow up for anyone looking here - this issue is [fixed](https://github.com/elastic/cloud-on-k8s/pull/3527) now and will be shipped in `1.2.1` release soon. We've also [added](https://github.com/elastic/cloud-on-k8s/pull/3541) additional comments directly in the yaml files and in the release notes. ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [November 4, 2022, 8:07am UTC](https://discuss.elastic.co/t/beats-deployment-error-on-eck-1-2-0/242260/9 "2022-11-04T08:07:20Z")