Hi @Paul_B
First, in the future, can you paste text, not screenshots of text much harder to work with...
Ok here is what I did...
I went and downloaded the filebeat for Kubernetes reference found here
This is always where I start from, or take a look if upgrading. This is the reference.
I updated to use the autodiscover out of the box....
Note: There is little change on the creds as mine are in secrets that I convert to ENV Vars
Note: I have no hints / annotations on any apps. (back to that later)
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
# filebeat.inputs:
# - type: filestream
# paths:
# - /var/log/containers/*.log
# parsers:
# - container: ~
# prospector:
# scanner:
# fingerprint.enabled: true
# symlinks: true
# file_identity.fingerprint: ~
# processors:
# - add_kubernetes_metadata:
# host: ${NODE_NAME}
# matchers:
# - logs_path:
# logs_path: "/var/log/containers/"
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
hints.default_config:
type: filestream
id: kubernetes-container-logs-${data.kubernetes.pod.name}-${data.kubernetes.container.id}
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
parsers:
- container: ~
prospector:
scanner:
fingerprint.enabled: true
symlinks: true
file_identity.fingerprint: ~
processors:
- add_cloud_metadata:
- add_host_metadata:
# cloud.id: ${ELASTIC_CLOUD_ID}
# cloud.auth: ${ELASTIC_CLOUD_AUTH}
output.elasticsearch:
hosts: ['${ES_HOST}']
username: ${ES_USERNAME}
password: ${ES_PASSWORD}
---
This works as expected it picks up all my apps, and when I kill them or restart them or delete them or redploy them logs continue to flow.
So I would start with that and then see if your annotations work...
Which speaking of ... I am not sure exactly how you are pulling those annotations because you did not show the command. My annotations do not have the app name in them
These annotations are for heartbeat autodiscover
hyperion:k8s sbrown$ kubectl get service my-otel-demo-frontend -o jsonpath='{.metadata.annotations}' | jq
{
"cloud.google.com/neg": "{\"ingress\":true,\"exposed_ports\":{\"8080\":{}}}",
"cloud.google.com/neg-status": "{\"network_endpoint_groups\":{\"8080\":\"k8s1-f6c1ea2c-default-my-otel-demo-frontend-8080-8604104a\"},\"zones\":[\"us-west2-a\",\"us-west2-b\",\"us-west2-c\"]}",
"co.elastic.monitor/hosts": "my-otel-demo-frontend.default.svc.cluster.local:8080",
"co.elastic.monitor/id": "my-otel-demo-frontend-monitor-id-1234",
"co.elastic.monitor/name": "my-otel-demo-frontend",
"co.elastic.monitor/schedule": "@every 10s",
"co.elastic.monitor/timeout": "5s",
"co.elastic.monitor/type": "http",
"meta.helm.sh/release-name": "my-otel-demo",
"meta.helm.sh/release-namespace": "default"
}
For filebeat the annotation should look as shown here
Not sure what yours are showing the service name in the path.
