Not able to access the filebeat custom fields in logstash output

I am trying to access an filebeat field in the logstash output. For some reason this is not working. Can someone help on this please?

Filebeat & logstash versions:
filebeat-6.4.1-1.x86_64
logstash-6.3.2-1.noarch

filebeat config:
- /var/log/nginx/access.log
fields:
type: access_log
test: testfield
tenant_id: XXXXXXXXXXXX
api_key: XXXXXXXXXXX
fields_under_root: true

lmm config:

input
{
beats
{
port => 5044
}
}
filter
{
if [type] == "cdg_access_log"
{
mutate
{
add_field => {
"tenant_id" => "%{[fields][tenant_id]}"
"api_key" => "%{[fields][api_key]}"
}
}
}
output
{
http
{
url => "XXXXXXX"
http_method => "post"
format => "json_batch"
headers => ["Content-Type", "application/json"]
headers => ["tenantid", "%{[fields][tenant_id]}"]
headers => ["apikey", "%{[fields][api_key]}"]
}
}

output event:

{:url=>"http://xxxxxxx", :method=>:post, :body=>"[{"@timestamp":"2018-12-04T10:17:30.306Z","type":"access_log","message":"55.255.0:80] [0] [60.048]\\n","@version":"1","tags":["aggregate"]}]", :headers=>{"Content-Type"=>"application/json", "tenantid"=>"%{[fields][tenant_id]}", "apikey"=>"%{[fields][api_key]}"}}

Look at the config examples here. Looks like you're calling the field names incorrectly. Also see link here for accessing field data.

Thanks Philip. So it always needs to be an metadata field?. I was going throught threads and refered this thread for my configuration

No. You're not working with metadata fields, just normal fields. Metadata fields are not added to output. You need something like this: [tenant_id] since you're declaring it to go under root.

Tried by changing the fields to metadata. Still the same issue

replaced with this in the filter section

mutate
{
add_field => ["[@metadata][lmmtenant]", "%{[fields][tenant_id]"]
add_field => ["[@metadata][lmmapikey]", "%{[fields][api_key]"]
}

Event output:
:headers=>{"Content-Type"=>"application/json", "tenantid"=>"%{[@metadata][lmmtenant]}", "apikey"=>"%{[@metadata][lmmapikey]}"}

I removed the under root option and did the following configuration

filter
{
mutate
{
add_field => ["tenant_id", "%{[fields][log_tenant_id]}"]
add_field => ["apikey", "%{[fields][log_api_key]}"]
}

}

output
{
http
{
url => "xxxxxxxxxxxx"
http_method => "post"
format => "json_batch"
headers => ["Content-Type", "application/json"]
headers => ["tenantid", "%{tenant_id}"]
headers => ["apikey", "%{apikey}"]
}
}

I can see the filebeat fields are getting assinged in the filter section. But not in the output section headers.

2018-12-04T19:27:51,397][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"tenant_id", "value"=>["%{[fields][log_tenant_id]}"]}

[2018-12-04T19:27:51,398][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"apikey", "value"=>["%{[fields][log_api_key]}"]}

[2018-12-04T19:27:51,409][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"tenant_id", "value"=>["%{[fields][log_tenant_id]}"]}

[2018-12-04T19:27:51,409][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"apikey", "value"=>["%{[fields][log_api_key]}"]}

[2018-12-04T19:27:51,479][DEBUG][logstash.pipeline ] output received {"event"=>{"host"=>{"name"=>"xxxxxxx"}, "fields"=>{"log_tenant_id"=>"xxxxxxx", "log_api_key"=>"xxxxxxxx", "type"=>"access_log"}, "@timestamp"=>2018-12-04T19:27:49.088Z, "tenant_id"=>"xxxxxxx", "offset"=>0, "input"=>{"type"=>"log"}, "apikey"=>"xxxxxxxxxx", "beat"=>{"hostname"=>"xxxxxxx", "version"=>"6.4.1", "name"=>"xxxxxx"}, "message"=>"127.0.0.1 - - [09/Oct/2018:14:28:30 +0000] "GET /1/config HTTP/1.1" 504 1 [504] [10.255.255.0:80] [0] [60.060]", "source"=>"/var/log/nginx/access.log", "prospector"=>{"type"=>"log"}, "tags"=>["beats_input_codec_plain_applied"], "@version"=>"1"}}

Hi Philip

Thanks a lot for your help and suggestions. The issue is with logstash http output format. After changing the format from "json_batch" to "json" , I am able to get the field value in the output.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.