# Kubernetes Heartbeat autodiscover not working

**URL:** <https://discuss.elastic.co/t/kubernetes-heartbeat-autodiscover-not-working/347867>\
**Category:** Beats\
**Tags:** heartbeat\
**Created:** [November 23, 2023, 2:33pm UTC](https://discuss.elastic.co/t/kubernetes-heartbeat-autodiscover-not-working/347867 "2023-11-23T14:33:35Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_b/32/48977_2.png) [@Paul\_B](https://discuss.elastic.co/u/Paul_B)\
**Post date:** [November 23, 2023, 2:33pm UTC](https://discuss.elastic.co/t/kubernetes-heartbeat-autodiscover-not-working/347867/1 "2023-11-23T14:33:35Z")

</div>

Hi all,

I've used the Kubernetes manifest file from this part of the documentation - [Running Heartbeat on Kubernetes | Heartbeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/heartbeat/current/running-on-kubernetes.html)

The deployment is working and the standalone monitors I've created are working, however, I'm also attempting to use the Autodiscover feature with pods annotated according to the documentation and they're all failing to start with the following error:

{  
"log.level": "warn",  
"@timestamp": "2023-11-23T11:20:52.889Z",  
"log.origin": {  
"file.name": "hints/monitors.go",  
"file.line": 123  
},  
"message": "unable to find valid hosts for {"name":"platform-admin","schedule":"@every 30s","type":"http","urls":"platform-admin.foundations.svc.cluster.local/health/ping"}: %!w(\*errors.errorString=&{no hosts selected for port 5000 with hints: })",  
"service.name": "heartbeat",  
"ecs.version": "1.6.0"  
}

This is an example of the set of annotations on each pod:

`co.elastic.monitor/name: platform-admin`  
`co.elastic.monitor/schedule: '@every 30s'`  
`co.elastic.monitor/type: http`  
`co.elastic.monitor/urls: platform-admin.foundations.svc.cluster.local/health/live`

Interestingly I also have some annotated pods that need a TCP monitor and they all work fine.

I've tried shelling into the heartbeat container and doing a curl request to the above HTTP health endpoint and I can get a 200 back so the pod/container itself has connectivity to the namespace in which the other service lives.

Any help would be appreciated.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 23, 2023, 4:38pm UTC](https://discuss.elastic.co/t/kubernetes-heartbeat-autodiscover-not-working/347867/2 "2023-11-23T16:38:51Z")

</div>

Hi @Paul_B Did you try specifying the port in the hints?

Also what scope are you using for the autodiscover?

I have a working version of this somewhere... I'll look over the weekend if you don't get it working.

Also am I missing something I think the annotation is

`co.elastic.monitor/hosts` I don't see `urls`

---

<div class="post-metadata">

**Author:** ![Paul\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_b/32/48977_2.png) [@Paul\_B](https://discuss.elastic.co/u/Paul_B)\
**Post date:** [November 23, 2023, 4:46pm UTC](https://discuss.elastic.co/t/kubernetes-heartbeat-autodiscover-not-working/347867/3 "2023-11-23T16:46:22Z")

</div>

@stephenb Ahh, I wasn't aware that there was a "port" hint, I'll try adding the following and see if it helps:

`co.elastic.monitor/port: 80`

I forgot to include my autodiscover config in the previous post:

```auto
heartbeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled
      node: ${NODE_NAME}
      scope: cluster
      resource: pod

```

With the above I've tried both `pod` and `service` for "resource" but neither made any difference.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 23, 2023, 5:05pm UTC](https://discuss.elastic.co/t/kubernetes-heartbeat-autodiscover-not-working/347867/4 "2023-11-23T17:05:48Z")

</div>

I think you want to use `service`... I am looking and a bit confused on `urls` vs `host` I see `urls` in some of my code. If you use `service` the annotation need to be on the service not the pod.

I am looking...

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 23, 2023, 6:01pm UTC](https://discuss.elastic.co/t/kubernetes-heartbeat-autodiscover-not-working/347867/5 "2023-11-23T18:01:31Z")

</div>

@Paul_B

Ok got it working...

BTW this is the sample Google Online Boutique App.

Now I am purposely monitoring the Service not the Pod .... I want to know when the service is down not 1 of the 3 pods if that makes sense...

```auto
data:
  heartbeat.yml: |-
    heartbeat.autodiscover: # Enable one or more of the providers below
      providers:
        - type: kubernetes
          resource: service
          scope: cluster
          node: ${NODE_NAME}
          hints.enabled: true

```

Here is a sample service yes it is `hosts` not `urls` I still had URLs in my code not sure if that changed at some point but `hosts` works `urls` does not.  
Also exec into the heartbeat container and make sure your urls resolve ...

```auto
apiVersion: v1
kind: Service
metadata:
  name: productcatalogservice
  namespace: product-catalog
  labels:
    project.skysthelimit.acme.ai/geography: eu
    project.skysthelimit.acme.ai/id: 3b3b229e-4de6-4496-bec8-06ff2eaff430
    project.skysthelimit.acme.ai/name: apikey-test
  annotations:
    co.elastic.monitor/id: "productcatalogservice-service-monitor-id-1234"
    co.elastic.monitor/type: "tcp"
    co.elastic.monitor/schedule: "@every 10s"
    co.elastic.monitor/timeout: "5s"
    co.elastic.monitor/name: "product-catalog-service"
    # co.elastic.monitor/urls: "productcatalogservice.product-catalog.svc.cluster.local:3550"
    co.elastic.monitor/hosts: "productcatalogservice.product-catalog.svc.cluster.local:3550"
    co.elastic.monitor/port: "3550"
    co.elastic.monitor/proc.0: "add_fields"
    co.elastic.monitor/proc.1: "add_geo"

spec:
  type: ClusterIP
  selector:
    app: productcatalogservice
  ports:
  - name: grpc
    port: 3550
    targetPort: 3550

```

Also you need to add

 ![Screenshot 2023-11-23 at 9.59.32 AM](https://us1.discourse-cdn.com/elastic/original/3X/1/3/1371143ef7f6b488299e0d309eb249e6cd0a12ef.png)

Happy Results

 ![Screenshot 2023-11-23 at 9.59.43 AM](https://us1.discourse-cdn.com/elastic/original/3X/0/8/0843006ead67ca184c9a38f7f305532cd27933ab.png)

---

<div class="post-metadata">

**Author:** ![Paul\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_b/32/48977_2.png) [@Paul\_B](https://discuss.elastic.co/u/Paul_B)\
**Post date:** [November 23, 2023, 6:41pm UTC](https://discuss.elastic.co/t/kubernetes-heartbeat-autodiscover-not-working/347867/6 "2023-11-23T18:41:49Z")

</div>

@stephenb brilliant! I think that might have sorted it 🙂

I'll add the annotation to a few more services and see how it goes. Also yes it makes perfect sense to target the services not the pods, thanks for the tip!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 21, 2023, 8:42pm UTC](https://discuss.elastic.co/t/kubernetes-heartbeat-autodiscover-not-working/347867/7 "2023-12-21T20:42:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.