# Help with parsing Kafka key

**URL:** <https://discuss.elastic.co/t/help-with-parsing-kafka-key/173542>\
**Category:** Logstash\
**Created:** [March 22, 2019, 5:36pm UTC](https://discuss.elastic.co/t/help-with-parsing-kafka-key/173542 "2019-03-22T17:36:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ed\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ed_k/32/42616_2.png) [@Ed\_K](https://discuss.elastic.co/u/Ed_K)\
**Post date:** [March 22, 2019, 5:36pm UTC](https://discuss.elastic.co/t/help-with-parsing-kafka-key/173542/1 "2019-03-22T17:36:52Z")

</div>

Hello, I currently use [@metadata][kafka][key] to set the document ID of all my messages from the kafka input. I want to implement a way to also route the message to the appropriate index. There are 2 options I am thinking of:

1. overload the key with "index\_name:document\_id" and parse this somehow in logstash
2. add the index name/pattern into the message itself as a field, and set the index with this field value

I was not able to find a resource on how to parse the key into multiple parts, and then reference the part back into the index/document\_id fields in the output, and I don't want to include the index name inside the document in elastic. I am ok with either approach if the solution is possible, or any suggestions on an alternate approach. Thanks!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 22, 2019, 8:44pm UTC](https://discuss.elastic.co/t/help-with-parsing-kafka-key/173542/2 "2019-03-22T20:44:50Z")

</div>

Just put the index name into [@metadata][index] and sprintf reference it in the output, very similar to what you are doing with the key.

---

<div class="post-metadata">

**Author:** ![Ed\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ed_k/32/42616_2.png) [@Ed\_K](https://discuss.elastic.co/u/Ed_K)\
**Post date:** [March 22, 2019, 11:39pm UTC](https://discuss.elastic.co/t/help-with-parsing-kafka-key/173542/3 "2019-03-22T23:39:41Z")

</div>

Thanks for the response, but I am not quite sure how I will set it into metadata. If i set it inside mutate, then I would need to fetch it from some source, such as the kafka key, but then I'd need to parse into tokens so I can extract the index name portion of the key. I don't know how to explicitly do this step (parsing the kafka key).

Or do you mean to say 'metadata' is a special keyword, and if I set a metadata field into the kafka message itself, I can get access to it?

e.g. { "field1" : "val1", "metadata" : { "index" : "idx\_pttrn" } }

I'm not too familiar with the metadata field. I just know that certain fields like kafka info are given to me automatically based on the decorate flag. But the index name has to be manually set by me somewhere, whether it comes from extracting as part of the kafka message key, or somewhere else.

Further example:

filter { mutate { add\_field =\> { "[@metadata][index]" =\> "[@metadata][kafka][key] \<~ how do i parse inside here????" } } }

---

<div class="post-metadata">

**Author:** ![Kim-Kruse-Hansen](https://avatars.discourse-cdn.com/v4/letter/k/f1d935/32.png) [@Kim-Kruse-Hansen](https://discuss.elastic.co/u/Kim-Kruse-Hansen)\
**Post date:** [March 23, 2019, 5:34pm UTC](https://discuss.elastic.co/t/help-with-parsing-kafka-key/173542/4 "2019-03-23T17:34:51Z")

</div>

Hi

I have written a blog post of how I do this. Have a read and see if you can use it

[https://www.securitydistractions.com/2019/03/23/simple-kafka-and-elasticsearch-integration/](https://www.securitydistractions.com/2019/03/23/simple-kafka-and-elasticsearch-integration/)

Kim

---

<div class="post-metadata">

**Author:** ![Ed\_K](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ed_k/32/42616_2.png) [@Ed\_K](https://discuss.elastic.co/u/Ed_K)\
**Post date:** [March 26, 2019, 6:19pm UTC](https://discuss.elastic.co/t/help-with-parsing-kafka-key/173542/5 "2019-03-26T18:19:04Z")

</div>

Thank you. I was hoping to make use of mutate/split of the kafka key, but there wasn't a good example. Instead I took the option of injecting an extra key into the message and appending that value to the index name. Here are some snippets:

```
...
filter {
  mutate { 
    add_field => ["[@metadata][idxSuffix]" , "%{suffix}" ] 
    remove_field => "suffix"
  }
}
...
output {
  elasticsearch {
    ...
    index => "myIndex-%{[@metadata][idxSuffix]}"
  }
}

```

I might revisit some metadata parsing eventually, but time constraints had me choosing a faster path 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 23, 2019, 6:19pm UTC](https://discuss.elastic.co/t/help-with-parsing-kafka-key/173542/6 "2019-04-23T18:19:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.