# Logstash pipeline for kafka message

**URL:** <https://discuss.elastic.co/t/logstash-pipeline-for-kafka-message/307413>\
**Category:** Logstash\
**Created:** [June 16, 2022, 1:24pm UTC](https://discuss.elastic.co/t/logstash-pipeline-for-kafka-message/307413 "2022-06-16T13:24:30Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![harijld](https://avatars.discourse-cdn.com/v4/letter/h/9fc348/32.png) [@harijld](https://discuss.elastic.co/u/harijld)\
**Post date:** [June 16, 2022, 1:24pm UTC](https://discuss.elastic.co/t/logstash-pipeline-for-kafka-message/307413/1 "2022-06-16T13:24:30Z")

</div>

Hi Team, Need some guidance to prepare log stash pipeline for below message. I am getting below message from KAFKA topic and i need to stash it into Elasticsearch though log stash. I am able to do it but I want to insert JSON fields as a separate field in Elasticsearch. not as a one complete message.

sample kafka message -  
{  
"request": {  
"cardnumber": "4545454545454554",  
"servicename": "ENQUIRY",  
"funcation": "VTSservice",  
"uniqueID": 124578545421,  
"application": "XYZ",  
"timestamp": "2020-05-21T15:13:18.853+04:00",  
"messagetrace": "tesing in progress",  
"status": "success"  
}  
}

CONFIG FILE IS -

```auto
input {
   kafka {
         bootstrap_servers => "localhost:9092"
        topics => "mytopic"

}
}

output {
elasticsearch {
hosts => ["XX.XXX.0.8:9200"]
index => "kafkalogs"
}
stdout { codec => rubydebug }
}

```

Elasticsearch snippet-

![image](https://us1.discourse-cdn.com/elastic/original/3X/c/0/c0de110569576a873c5ff359e27c47748e360e6f.png)

I want it display / stash my message in individual field in Elasticsearch.

I want help in preparing config file with filters..

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 16, 2022, 3:50pm UTC](https://discuss.elastic.co/t/logstash-pipeline-for-kafka-message/307413/2 "2022-06-16T15:50:23Z")

</div>

Use a [json](https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) filter.

---

<div class="post-metadata">

**Author:** ![harijld](https://avatars.discourse-cdn.com/v4/letter/h/9fc348/32.png) [@harijld](https://discuss.elastic.co/u/harijld)\
**Post date:** [June 29, 2022, 9:16am UTC](https://discuss.elastic.co/t/logstash-pipeline-for-kafka-message/307413/3 "2022-06-29T09:16:49Z")

</div>

HI @badger, Let me ask in another way.  
I have created index in Elasticsearch with 5 fields. like field1, field2.....field5.  
In input Json file is also having 5 fields. field A, field B, field C..... Field E.

Now i want to map field A to field 1 in perticuler index. and here i want to do some data conversion.

how can map the input json fields to index fields.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 29, 2022, 4:34pm UTC](https://discuss.elastic.co/t/logstash-pipeline-for-kafka-message/307413/4 "2022-06-29T16:34:25Z")

</div>

You can use a mutate filter to rename fields and to do type conversions.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 27, 2022, 4:34pm UTC](https://discuss.elastic.co/t/logstash-pipeline-for-kafka-message/307413/5 "2022-07-27T16:34:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.