# Reading beats fields from Kafka input

**URL:** https://discuss.elastic.co/t/reading-beats-fields-from-kafka-input/76206
**Category:** Logstash
**Created:** [February 23, 2017, 11:43am UTC](https://discuss.elastic.co/t/reading-beats-fields-from-kafka-input/76206 "2017-02-23T11:43:52Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Rober](https://avatars.discourse-cdn.com/v4/letter/r/48db29/32.png) [@Rober](https://discuss.elastic.co/u/Rober)
#### Post date: [February 23, 2017, 11:43am UTC](https://discuss.elastic.co/t/reading-beats-fields-from-kafka-input/76206/1 "2017-02-23T11:43:52Z")

</div>

Hi,  
in my infrastructure I have something like:  
Filebeat -\> Kafka -\> Logstash -\> Elasticsearch

I'm having problems reading the custom fields set up in Filebeat when filtering in Logstash.

I configured the kafka input in Logstash, I'd like to be able to read content of "my\_custom\_field", but I'm unable to...

```
{
	"message" : {
		"@timestamp":"2017-02-22T17:13:22.346Z",
		"beat":{
			"hostname":"host-01",
			"name":"name-01",
			"version":"5.2.1"},
			"fields":{
				"my_custom_field":"XXXXX"
			},
		"input_type":"log",
		"message":"....................",
		"offset":6894303,
		"source":"...",
		"type":"log"
	}
}

```

How can I access this field in order to create my index name (output) using it?

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [February 27, 2017, 6:41am UTC](https://discuss.elastic.co/t/reading-beats-fields-from-kafka-input/76206/2 "2017-02-27T06:41:49Z")

</div>

I'm confused as to why all fields are nested under `message`, but if this indeed is what your event looks like you can access `my_custom_field` with `[message][beat][fields][my_custom_field]`.

[https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#logstash-config-field-references)

---

<div class="post-metadata">

### Author: ![Rober](https://avatars.discourse-cdn.com/v4/letter/r/48db29/32.png) [@Rober](https://discuss.elastic.co/u/Rober)
#### Post date: [February 27, 2017, 11:57am UTC](https://discuss.elastic.co/t/reading-beats-fields-from-kafka-input/76206/3 "2017-02-27T11:57:10Z")

</div>

Thanks!  
Actually, what I'm doing now to get this field is:

```
json {
	source => "message"
	target => "beat_details"
}	
mutate { 
	add_field => { "type" => "%{[beat_details][fields][my_custom_field]}" } 	
}	

```

Then I can use this "type" in my filters.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 27, 2017, 11:57am UTC](https://discuss.elastic.co/t/reading-beats-fields-from-kafka-input/76206/4 "2017-03-27T11:57:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.