# Logstash Adding Filebeat Fields

**URL:** <https://discuss.elastic.co/t/logstash-adding-filebeat-fields/149538>\
**Category:** Logstash\
**Created:** [September 22, 2018, 10:17am UTC](https://discuss.elastic.co/t/logstash-adding-filebeat-fields/149538 "2018-09-22T10:17:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![eswarloges](https://avatars.discourse-cdn.com/v4/letter/e/8e7dd6/32.png) [@eswarloges](https://discuss.elastic.co/u/eswarloges)\
**Post date:** [September 22, 2018, 10:17am UTC](https://discuss.elastic.co/t/logstash-adding-filebeat-fields/149538/1 "2018-09-22T10:17:29Z")

</div>

Hi Team,

The design we used to read log files - Filebeat -\> Kafka Topic -\> Logstash -\> Elasticsearch

When the message is arriving in Logstash it is in the following format

> {  
> "@timestamp":"2018-09-20T00:23:48.543Z",
> 
> "@metadata":{  
> "beat":"filebeat",  
> "type":"doc",  
> "version":"6.3.2",  
> "topic":"kafka\_broker\_1"  
> },  
> "source":"/apps/kafka/confluent-4.0.0/logs/server.log",  
> "offset":5158,  
> "message":"[2018-09-19 19:23:46,612] INFO Incrementing log start offset of partition \_confluent-metrics-5 to 8644625 in dir /data/kafka/kafka-logs (kafka.log.Log)",  
> "prospector":{  
> "type":"log"  
> },  
> "input":{  
> "type":"log"  
> },  
> "beat":{  
> "name":"test01",  
> "hostname":"test01",  
> "version":"6.3.2"  
> },  
> "host":{  
> "name":"test01"  
> }  
> }

And I m using the following filter in Logstash

> filter {
> 
> ```
> mutate {
> 
> add_field => {
> "source" => "%{[message][source]}"
> "kafkaTopic" => "%{[@metadata][kafka][topic]}"
> }
> }
> 
> grok {
> match => { "inputjson" => "\[%{TIMESTAMP_ISO8601:logTime}\] %{LOGLEVEL:severity} %{GREEDYDATA:logMessage} \(%{JAVACLASS:loggerName}\)"}
> 
> }
> 
> date {
> match => ["logTime", "yyyy-MM-dd HH:mm:ss,SSS"]
> target => "logTime"
> }
> 
> if "_grokparsefailure" in [tags] {
> mutate {
> remove_field => ["@version","path","type","host"]
> }
> } else {
> mutate {
> remove_field => ["message","@version","path","type","host"]
> }
> }
> 
> ```
> 
> }

And the source filed is coming as string in the ES

> "source" : "%{[message][source]}"

How to access the nested fields in the incoming message? Any help/directions would help me to fix the issue.Thanks

Regards,  
Logeswaran Radhakrishnan

---

<div class="post-metadata">

**Author:** ![eswarloges](https://avatars.discourse-cdn.com/v4/letter/e/8e7dd6/32.png) [@eswarloges](https://discuss.elastic.co/u/eswarloges)\
**Post date:** [September 24, 2018, 8:32am UTC](https://discuss.elastic.co/t/logstash-adding-filebeat-fields/149538/2 "2018-09-24T08:32:27Z")

</div>

The reason I want to extract the fields like beat name, beat hostname and source because it is being dropped by the Logstash.

I have added the json codec and the fields are getting indexed properly. Thanks

This topic can be closed.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 25, 2018, 9:36am UTC](https://discuss.elastic.co/t/logstash-adding-filebeat-fields/149538/3 "2018-09-25T09:36:18Z")

</div>

> How to access the nested fields in the incoming message?

Which nested fields? `source` isn't nested and the Kafka topic field is clearly named `[@metadata][topic]`.

---

<div class="post-metadata">

**Author:** ![eswarloges](https://avatars.discourse-cdn.com/v4/letter/e/8e7dd6/32.png) [@eswarloges](https://discuss.elastic.co/u/eswarloges)\
**Post date:** [September 25, 2018, 10:09am UTC](https://discuss.elastic.co/t/logstash-adding-filebeat-fields/149538/4 "2018-09-25T10:09:38Z")

</div>

@magnusbaeck  
I was trying to accessing fields like beat name and beat hostname. The above code representation was just representing the source .

The patterns like %{[message][beat][name]} , %{[message][source]} nothing was working.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 25, 2018, 11:08am UTC](https://discuss.elastic.co/t/logstash-adding-filebeat-fields/149538/5 "2018-09-25T11:08:50Z")

</div>

None of those fields are nested under `message`. Use `[beat][name]`, `source`, and so on.

---

<div class="post-metadata">

**Author:** ![eswarloges](https://avatars.discourse-cdn.com/v4/letter/e/8e7dd6/32.png) [@eswarloges](https://discuss.elastic.co/u/eswarloges)\
**Post date:** [October 1, 2018, 10:17am UTC](https://discuss.elastic.co/t/logstash-adding-filebeat-fields/149538/6 "2018-10-01T10:17:27Z")

</div>

Working fine. Thanks @magnusbaeck

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 29, 2018, 10:17am UTC](https://discuss.elastic.co/t/logstash-adding-filebeat-fields/149538/7 "2018-10-29T10:17:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.