Hello,
I am trying to configue an integration between airflow and kibana using Airflow | Documentation (elastic.co). I deployed elasticsearch/kibana/agent/fleet-server using ECK operator.
here is kibana yaml:
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana
namespace: elasticapm
spec:
version: 8.13.2
count: 1
elasticsearchRef:
name: elasticsearch
config:
xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.elasticapm.svc:9200"]
xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.elasticapm.svc:8220"]
xpack.fleet.packages:
- name: system
version: latest
- name: elastic_agent
version: latest
- name: fleet_server
version: latest
- name: airflow
version: 0.6.0
xpack.fleet.agentPolicies:
- name: Fleet Server on ECK policy
id: eck-fleet-server
namespace: default
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
package_policies:
- name: fleet_server-1
id: fleet_server-1
package:
name: fleet_server
- name: Elastic Agent on ECK policy
id: eck-agent
namespace: default
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
package_policies:
- name: system-1
id: system-1
package:
name: system
- name: airflow-1
id: airflow1-1
package:
name: airflow
fleet-server.yaml:
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: fleet-server
namespace: elasticapm
spec:
version: 8.13.2
kibanaRef:
name: kibana
elasticsearchRefs:
- name: elasticsearch
mode: fleet
fleetServerEnabled: true
policyID: eck-fleet-server
deployment:
replicas: 1
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
containers:
- name: agent
env:
- name: FLEET_SERVER_ELASTICSEARCH_CA
value: /etc/pki/tls/certs/ca.crt
volumeMounts:
- name: elasticsearch-certs
readOnly: true
mountPath: /etc/pki/tls/certs/ca.crt
subPath: ca.crt
volumes:
- name: agent-data
emptyDir: {}
agent.yaml:
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: elastic
namespace: elasticapm
spec:
version: 8.13.2
kibanaRef:
name: kibana
fleetServerRef:
name: fleet-server
mode: fleet
policyID: eck-agent
daemonSet:
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
containers:
- name: agent
env:
- name: FLEET_SERVER_ELASTICSEARCH_CA
value: /etc/pki/tls/certs/ca.crt
volumeMounts:
- name: elasticsearch-certs
readOnly: true
mountPath: /etc/pki/tls/certs/ca.crt
subPath: ca.crt
ports:
- containerPort: 8125
protocol: UDP
volumes:
- name: agent-data
emptyDir: {}
Agent reports to kibana, everything looks fine.
In the agent log I see:
{"log.level":"info","@timestamp":"2024-04-19T19:22:56.117Z","message":"Started listening for UDP on: 0.0.0.0:8125","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"statsd/metrics-default","type":"statsd/metrics"},"log":{"source":"statsd/metrics-default"},"log.origin":{"file.line":82,"file.name":"udp/udp.go","function":"github.com/elastic/beats/v7/metricbeat/helper/server/udp.(*UdpServer).Start"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
Airflow metrrics dashboard is empty and in the metrics explorer I don't see any airflow related metrics. I also tried to send a fake metric using netcat like this:
echo "foo:1|c" | nc -u -w0 <agent_ip> 8125
and didn't get the foo metric stored. I don't see error messages in the agent's log. Can anyone help me to figure out why it doesn't work?
