# Kafka Module pipeline is not working

**URL:** <https://discuss.elastic.co/t/kafka-module-pipeline-is-not-working/219351>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 14, 2020, 10:37am UTC](https://discuss.elastic.co/t/kafka-module-pipeline-is-not-working/219351 "2020-02-14T10:37:54Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![maewonline](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@maewonline](https://discuss.elastic.co/u/maewonline)\
**Post date:** [February 14, 2020, 10:37am UTC](https://discuss.elastic.co/t/kafka-module-pipeline-is-not-working/219351/1 "2020-02-14T10:37:55Z")

</div>

I want to send Kafka logs to Elastic through Logstash. I have configured the filebeat.yml file and I'm able to see my logs in Kibana but without the Kafka exported fields that applied using Kafka module pipeline.json

Filebeat Version: 6.7.0

**Filebeat.yml**

```
filebeat.config.modules:
  enabled: true
  path: /software/tools/filebeat/modules.d/*.yml

filebeat.modules:
- module: kafka
  log:
    enabled: true
    var.kafka_home: "/opt/kafka"
    var.paths:
      - "/logs/kafka/controller.log*"
      - "/logs/kafka/server.log*"
      - "/logs/kafka/state-change.log*"
      - "/logs/kafka/kafka-*.log*"
    fields:
      event_environment_id: fdt2
    fields_under_root: true
    scan_frequency: 1s
    max_bytes: 52428800

output.logstash:
  hosts: ["elkhost1:9285","elkhost2:9285","elkhost3:9285","elkhost4:9285","elkhost5:9285","elkhost6:9285","elkhost7:9285","elkhost8:9285"]
  loadbalance: true
  ttl: 60
  bulk_max_size: 256
  slow_start: true

```

Here how is the logs looks like in Kibana:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/7/f7134ea1e761d51669dd265a51104afe59e18fc2.png)

Here is the Kafka exported fields as per the following documentation:  
[https://www.elastic.co/guide/en/beats/filebeat/master/exported-fields-kafka.html](https://www.elastic.co/guide/en/beats/filebeat/master/exported-fields-kafka.html)

So why i can't see these fields? Am I missing any attributes in the filebeat configuration?

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [February 19, 2020, 1:46pm UTC](https://discuss.elastic.co/t/kafka-module-pipeline-is-not-working/219351/2 "2020-02-19T13:46:12Z")

</div>

Hi,

Given your Filebeat configuration, I'm wondering if the Filebeat index template has been installed properly. Could you please post the results of the following Elasticsearch API call?

```auto
GET _cat/templates/f*

```

Thanks,

Shaunak

---

<div class="post-metadata">

**Author:** ![maewonline](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@maewonline](https://discuss.elastic.co/u/maewonline)\
**Post date:** [February 25, 2020, 9:55am UTC](https://discuss.elastic.co/t/kafka-module-pipeline-is-not-working/219351/3 "2020-02-25T09:55:55Z")

</div>

Thanks Shaunak for your important reply. Unfortunately, I don't have access to ingest the Kafka pipeline configuration using Elasticsearch API.

So to solve this issue I have added all the logic exist in the below json file inside filter section of my logstash configuration file  
\<FILE\_BEAT\_INSTALL\>/module/kafka/log/ingest/pipeline.json

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 24, 2020, 9:56am UTC](https://discuss.elastic.co/t/kafka-module-pipeline-is-not-working/219351/4 "2020-03-24T09:56:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 24, 2020, 9:56am UTC](https://discuss.elastic.co/t/kafka-module-pipeline-is-not-working/219351/5 "2020-03-24T09:56:02Z")

</div>



---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 7:45am UTC](https://discuss.elastic.co/t/kafka-module-pipeline-is-not-working/219351/6 "2022-11-04T07:45:03Z")

</div>