# Not able to access the filebeat custom fields in logstash output

**URL:** <https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342>\
**Category:** Logstash\
**Created:** [December 4, 2018, 10:54am UTC](https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342 "2018-12-04T10:54:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ramalingam\_Chandran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramalingam_chandran/32/38420_2.png) [@Ramalingam\_Chandran](https://discuss.elastic.co/u/Ramalingam_Chandran)\
**Post date:** [December 4, 2018, 10:54am UTC](https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342/1 "2018-12-04T10:54:51Z")

</div>

I am trying to access an filebeat field in the logstash output. For some reason this is not working. Can someone help on this please?

Filebeat & logstash versions:  
filebeat-6.4.1-1.x86\_64  
logstash-6.3.2-1.noarch

filebeat config:  
- /var/log/nginx/access.log  
fields:  
type: access\_log  
test: testfield  
tenant\_id: XXXXXXXXXXXX  
api\_key: XXXXXXXXXXX  
fields\_under\_root: true

lmm config:

input  
{  
beats  
{  
port =\> 5044  
}  
}  
filter  
{  
if [type] == "cdg\_access\_log"  
{  
mutate  
{  
add\_field =\> {  
"tenant\_id" =\> "%{[fields][tenant\_id]}"  
"api\_key" =\> "%{[fields][api\_key]}"  
}  
}  
}  
output  
{  
http  
{  
url =\> "XXXXXXX"  
http\_method =\> "post"  
format =\> "json\_batch"  
headers =\> ["Content-Type", "application/json"]  
headers =\> ["tenantid", "%{[fields][tenant\_id]}"]  
headers =\> ["apikey", "%{[fields][api\_key]}"]  
}  
}

output event:

{:url=\>"[http://xxxxxxx](http://xxxxxxx)", :method=\>:post, :body=\>"[{"@timestamp":"2018-12-04T10:17:30.306Z","type":"access\_log","message":"55.255.0:80] [0] [60.048]\\n","@version":"1","tags":["aggregate"]}]", :headers=\>{"Content-Type"=\>"application/json", "tenantid"=\>"%{[fields][tenant\_id]}", "apikey"=\>"%{[fields][api\_key]}"}}

---

<div class="post-metadata">

**Author:** ![bigphil](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bigphil](https://discuss.elastic.co/u/bigphil)\
**Post date:** [December 4, 2018, 5:41pm UTC](https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342/2 "2018-12-04T17:41:26Z")

</div>

Look at the config examples [here](https://www.elastic.co/guide/en/logstash/6.5/logstash-config-for-filebeat-modules.html#parsing-nginx). Looks like you're calling the field names incorrectly. Also see link [here](https://www.elastic.co/guide/en/logstash/6.5/event-dependent-configuration.html) for accessing field data.

---

<div class="post-metadata">

**Author:** ![Ramalingam\_Chandran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramalingam_chandran/32/38420_2.png) [@Ramalingam\_Chandran](https://discuss.elastic.co/u/Ramalingam_Chandran)\
**Post date:** [December 4, 2018, 5:58pm UTC](https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342/3 "2018-12-04T17:58:02Z")

</div>

Thanks Philip. So it always needs to be an metadata field?. I was going throught threads and refered this thread for my configuration

> [@Filebeat fields are not working](https://discuss.elastic.co/t/filebeat-fields-are-not-working/122155):
>
> Here is my filebeat config: filebeat.prospectors: - type: log paths: - C:/inetpub/app01/logs/IIS/\*/\*.log fields: app\_group: iis app\_id: app01 - type: log paths: - C:/inetpub/app02/logs/IIS/\*/\*.log fields: app\_group: iis app\_id: app02 - type: log paths: - C:/inetpub/app03/logs/IIS/\*/\*.log fields: app\_group: iis app\_id: app03 output: logstash: hosts: ["myserver:7777"] And here is logstash: input { beats { port =\> 7777 } } filter …

---

<div class="post-metadata">

**Author:** ![bigphil](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bigphil](https://discuss.elastic.co/u/bigphil)\
**Post date:** [December 4, 2018, 6:41pm UTC](https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342/4 "2018-12-04T18:41:26Z")

</div>

No. You're not working with metadata fields, just normal fields. Metadata fields are not added to output. You need something like this: [tenant\_id] since you're declaring it to go under root.

---

<div class="post-metadata">

**Author:** ![Ramalingam\_Chandran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramalingam_chandran/32/38420_2.png) [@Ramalingam\_Chandran](https://discuss.elastic.co/u/Ramalingam_Chandran)\
**Post date:** [December 4, 2018, 6:41pm UTC](https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342/5 "2018-12-04T18:41:31Z")

</div>

Tried by changing the fields to metadata. Still the same issue

replaced with this in the filter section

mutate  
{  
add\_field =\> ["[@metadata][lmmtenant]", "%{[fields][tenant\_id]"]  
add\_field =\> ["[@metadata][lmmapikey]", "%{[fields][api\_key]"]  
}

Event output:  
:headers=\>{"Content-Type"=\>"application/json", "tenantid"=\>"%{[@metadata][lmmtenant]}", "apikey"=\>"%{[@metadata][lmmapikey]}"}

---

<div class="post-metadata">

**Author:** ![Ramalingam\_Chandran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramalingam_chandran/32/38420_2.png) [@Ramalingam\_Chandran](https://discuss.elastic.co/u/Ramalingam_Chandran)\
**Post date:** [December 4, 2018, 7:50pm UTC](https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342/6 "2018-12-04T19:50:35Z")

</div>

I removed the under root option and did the following configuration

filter  
{  
mutate  
{  
add\_field =\> ["tenant\_id", "%{[fields][log\_tenant\_id]}"]  
add\_field =\> ["apikey", "%{[fields][log\_api\_key]}"]  
}

}

output  
{  
http  
{  
url =\> "xxxxxxxxxxxx"  
http\_method =\> "post"  
format =\> "json\_batch"  
headers =\> ["Content-Type", "application/json"]  
headers =\> ["tenantid", "%{tenant\_id}"]  
headers =\> ["apikey", "%{apikey}"]  
}  
}

I can see the filebeat fields are getting assinged in the filter section. But not in the output section headers.

2018-12-04T19:27:51,397][DEBUG][logstash.util.decorators] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"tenant\_id", "value"=\>["%{[fields][log\_tenant\_id]}"]}

[2018-12-04T19:27:51,398][DEBUG][logstash.util.decorators] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"apikey", "value"=\>["%{[fields][log\_api\_key]}"]}

[2018-12-04T19:27:51,409][DEBUG][logstash.util.decorators] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"tenant\_id", "value"=\>["%{[fields][log\_tenant\_id]}"]}

[2018-12-04T19:27:51,409][DEBUG][logstash.util.decorators] filters/LogStash::Filters::Mutate: adding value to field {"field"=\>"apikey", "value"=\>["%{[fields][log\_api\_key]}"]}

[2018-12-04T19:27:51,479][DEBUG][logstash.pipeline] output received {"event"=\>{"host"=\>{"name"=\>"xxxxxxx"}, "fields"=\>{"log\_tenant\_id"=\>"xxxxxxx", "log\_api\_key"=\>"xxxxxxxx", "type"=\>"access\_log"}, "@timestamp"=\>2018-12-04T19:27:49.088Z, "tenant\_id"=\>"xxxxxxx", "offset"=\>0, "input"=\>{"type"=\>"log"}, "apikey"=\>"xxxxxxxxxx", "beat"=\>{"hostname"=\>"xxxxxxx", "version"=\>"6.4.1", "name"=\>"xxxxxx"}, "message"=\>"127.0.0.1 - - [09/Oct/2018:14:28:30 +0000] "GET /1/config HTTP/1.1" 504 1 [504] [10.255.255.0:80] [0] [60.060]", "source"=\>"/var/log/nginx/access.log", "prospector"=\>{"type"=\>"log"}, "tags"=\>["beats\_input\_codec\_plain\_applied"], "@version"=\>"1"}}

---

<div class="post-metadata">

**Author:** ![Ramalingam\_Chandran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramalingam_chandran/32/38420_2.png) [@Ramalingam\_Chandran](https://discuss.elastic.co/u/Ramalingam_Chandran)\
**Post date:** [December 5, 2018, 6:22am UTC](https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342/7 "2018-12-05T06:22:04Z")

</div>

Hi Philip

Thanks a lot for your help and suggestions. The issue is with logstash http output format. After changing the format from "json\_batch" to "json" , I am able to get the field value in the output.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 2, 2019, 6:22am UTC](https://discuss.elastic.co/t/not-able-to-access-the-filebeat-custom-fields-in-logstash-output/159342/8 "2019-01-02T06:22:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.