# Airflow integration doesn't work properly

**URL:** <https://discuss.elastic.co/t/airflow-integration-doesnt-work-properly/357823>\
**Category:** Elastic Observability\
**Created:** [April 20, 2024, 12:17am UTC](https://discuss.elastic.co/t/airflow-integration-doesnt-work-properly/357823 "2024-04-20T00:17:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![IvanovOleg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanovoleg/32/89473_2.png) [@IvanovOleg](https://discuss.elastic.co/u/IvanovOleg)\
**Post date:** [April 20, 2024, 12:17am UTC](https://discuss.elastic.co/t/airflow-integration-doesnt-work-properly/357823/1 "2024-04-20T00:17:19Z")

</div>

Hello,  
I am trying to configue an integration between airflow and kibana using [Airflow | Documentation (elastic.co)](https://docs.elastic.co/integrations/airflow). I deployed elasticsearch/kibana/agent/fleet-server using ECK operator.  
here is kibana yaml:

```auto
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elasticapm
spec:
  version: 8.13.2
  count: 1
  elasticsearchRef:
    name: elasticsearch
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.elasticapm.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.elasticapm.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
      - name: airflow
        version: 0.6.0
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system
          - name: airflow-1
            id: airflow1-1
            package:
              name: airflow

```

fleet-server.yaml:

```auto
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
  namespace: elasticapm
spec:
  version: 8.13.2
  kibanaRef:
    name: kibana
  elasticsearchRefs:
    - name: elasticsearch
  mode: fleet
  fleetServerEnabled: true
  policyID: eck-fleet-server
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        containers:
          - name: agent
            env:
              - name: FLEET_SERVER_ELASTICSEARCH_CA
                value: /etc/pki/tls/certs/ca.crt
            volumeMounts:
              - name: elasticsearch-certs
                readOnly: true
                mountPath: /etc/pki/tls/certs/ca.crt
                subPath: ca.crt
        volumes:
          - name: agent-data
            emptyDir: {}

```

agent.yaml:

```auto
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic
  namespace: elasticapm
spec:
  version: 8.13.2
  kibanaRef:
    name: kibana
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        containers:
          - name: agent
            env:
              - name: FLEET_SERVER_ELASTICSEARCH_CA
                value: /etc/pki/tls/certs/ca.crt
            volumeMounts:
              - name: elasticsearch-certs
                readOnly: true
                mountPath: /etc/pki/tls/certs/ca.crt
                subPath: ca.crt
            ports:
              - containerPort: 8125
                protocol: UDP
        volumes:
          - name: agent-data
            emptyDir: {}

```

Agent reports to kibana, everything looks fine.  
In the agent log I see:

```auto
{"log.level":"info","@timestamp":"2024-04-19T19:22:56.117Z","message":"Started listening for UDP on: 0.0.0.0:8125","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"statsd/metrics-default","type":"statsd/metrics"},"log":{"source":"statsd/metrics-default"},"log.origin":{"file.line":82,"file.name":"udp/udp.go","function":"github.com/elastic/beats/v7/metricbeat/helper/server/udp.(*UdpServer).Start"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

```

Airflow metrrics dashboard is empty and in the metrics explorer I don't see any airflow related metrics. I also tried to send a fake metric using netcat like this:

```auto
echo "foo:1|c" | nc -u -w0 <agent_ip> 8125

```

and didn't get the foo metric stored. I don't see error messages in the agent's log. Can anyone help me to figure out why it doesn't work?

 ![integration](https://us1.discourse-cdn.com/elastic/original/3X/c/8/c8daae3c6126f8ceb05368e3ff9db2a30898b9fc.png)  
 ![agent](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af3c8440c4051dd63f5532a80a5d1da7230f34bd.png)

---

<div class="post-metadata">

**Author:** ![georgegach](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/georgegach/32/139807_2.png) [@georgegach](https://discuss.elastic.co/u/georgegach)\
**Post date:** [December 9, 2024, 7:52am UTC](https://discuss.elastic.co/t/airflow-integration-doesnt-work-properly/357823/2 "2024-12-09T07:52:06Z")

</div>

Hey @IvanovOleg

Did you have any luck fixing the issue?

---

<div class="post-metadata">

**Author:** ![IvanovOleg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivanovoleg/32/89473_2.png) [@IvanovOleg](https://discuss.elastic.co/u/IvanovOleg)\
**Post date:** [December 9, 2024, 3:01pm UTC](https://discuss.elastic.co/t/airflow-integration-doesnt-work-properly/357823/3 "2024-12-09T15:01:34Z")

</div>

Hello. Take a look here. [Components stuck waiting on migrations · Issue #38937 · apache/airflow · GitHub](https://github.com/apache/airflow/issues/38937)