Kubernetes Heartbeat autodiscover not working

[Paul_B (Paul)]

Hi all,

I've used the Kubernetes manifest file from this part of the documentation - Running Heartbeat on Kubernetes | Heartbeat Reference [8.11] | Elastic

The deployment is working and the standalone monitors I've created are working, however, I'm also attempting to use the Autodiscover feature with pods annotated according to the documentation and they're all failing to start with the following error:

{
"log.level": "warn",
"@timestamp": "2023-11-23T11:20:52.889Z",
"log.origin": {
"file.name": "hints/monitors.go",
"file.line": 123
},
"message": "unable to find valid hosts for {"name":"platform-admin","schedule":"@every 30s","type":"http","urls":"platform-admin.foundations.svc.cluster.local/health/ping"}: %!w(*errors.errorString=&{no hosts selected for port 5000 with hints: })",
"service.name": "heartbeat",
"ecs.version": "1.6.0"
}

This is an example of the set of annotations on each pod:

co.elastic.monitor/name: platform-admin
co.elastic.monitor/schedule: '@every 30s'
co.elastic.monitor/type: http
co.elastic.monitor/urls: platform-admin.foundations.svc.cluster.local/health/live

Interestingly I also have some annotated pods that need a TCP monitor and they all work fine.

I've tried shelling into the heartbeat container and doing a curl request to the above HTTP health endpoint and I can get a 200 back so the pod/container itself has connectivity to the namespace in which the other service lives.

Any help would be appreciated.

[stephenb (Stephen Brown)]

Hi @Paul_B Did you try specifying the port in the hints?

Also what scope are you using for the autodiscover?

I have a working version of this somewhere... I'll look over the weekend if you don't get it working.

Also am I missing something I think the annotation is

co.elastic.monitor/hosts I don't see urls

[Paul_B (Paul)]

@stephenb Ahh, I wasn't aware that there was a "port" hint, I'll try adding the following and see if it helps:

co.elastic.monitor/port: 80

I forgot to include my autodiscover config in the previous post:

heartbeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled
      node: ${NODE_NAME}
      scope: cluster
      resource: pod

With the above I've tried both pod and service for "resource" but neither made any difference.

[stephenb (Stephen Brown)]

I think you want to use service... I am looking and a bit confused on urls vs host I see urls in some of my code. If you use service the annotation need to be on the service not the pod.

I am looking...

[stephenb (Stephen Brown)]

@Paul_B

Ok got it working...

BTW this is the sample Google Online Boutique App.

Now I am purposely monitoring the Service not the Pod .... I want to know when the service is down not 1 of the 3 pods if that makes sense...

data:
  heartbeat.yml: |-
    heartbeat.autodiscover: # Enable one or more of the providers below
      providers:
        - type: kubernetes
          resource: service
          scope: cluster
          node: ${NODE_NAME}
          hints.enabled: true

Here is a sample service yes it is hosts not urls I still had URLs in my code not sure if that changed at some point but hosts works urls does not.
Also exec into the heartbeat container and make sure your urls resolve ...

apiVersion: v1
kind: Service
metadata:
  name: productcatalogservice
  namespace: product-catalog
  labels:
    project.skysthelimit.acme.ai/geography: eu
    project.skysthelimit.acme.ai/id: 3b3b229e-4de6-4496-bec8-06ff2eaff430
    project.skysthelimit.acme.ai/name: apikey-test
  annotations:
    co.elastic.monitor/id: "productcatalogservice-service-monitor-id-1234"
    co.elastic.monitor/type: "tcp"
    co.elastic.monitor/schedule: "@every 10s"
    co.elastic.monitor/timeout: "5s"
    co.elastic.monitor/name: "product-catalog-service"
    # co.elastic.monitor/urls: "productcatalogservice.product-catalog.svc.cluster.local:3550"
    co.elastic.monitor/hosts: "productcatalogservice.product-catalog.svc.cluster.local:3550"
    co.elastic.monitor/port: "3550"
    co.elastic.monitor/proc.0: "add_fields"
    co.elastic.monitor/proc.1: "add_geo"


spec:
  type: ClusterIP
  selector:
    app: productcatalogservice
  ports:
  - name: grpc
    port: 3550
    targetPort: 3550

Also you need to add

Happy Results

[Paul_B (Paul)]

@stephenb brilliant! I think that might have sorted it

I'll add the annotation to a few more services and see how it goes. Also yes it makes perfect sense to target the services not the pods, thanks for the tip!