Not able to access the filebeat custom fields in logstash output

[Ramalingam_Chandran]

I am trying to access an filebeat field in the logstash output. For some reason this is not working. Can someone help on this please?

Filebeat & logstash versions:
filebeat-6.4.1-1.x86_64
logstash-6.3.2-1.noarch

filebeat config:
- /var/log/nginx/access.log
fields:
type: access_log
test: testfield
tenant_id: XXXXXXXXXXXX
api_key: XXXXXXXXXXX
fields_under_root: true

lmm config:

input
{
beats
{
port => 5044
}
}
filter
{
if [type] == "cdg_access_log"
{
mutate
{
add_field => {
"tenant_id" => "%{[fields][tenant_id]}"
"api_key" => "%{[fields][api_key]}"
}
}
}
output
{
http
{
url => "XXXXXXX"
http_method => "post"
format => "json_batch"
headers => ["Content-Type", "application/json"]
headers => ["tenantid", "%{[fields][tenant_id]}"]
headers => ["apikey", "%{[fields][api_key]}"]
}
}

output event:

{:url=>"http://xxxxxxx", :method=>:post, :body=>"[{"@timestamp":"2018-12-04T10:17:30.306Z","type":"access_log","message":"55.255.0:80] [0] [60.048]\\n","@version":"1","tags":["aggregate"]}]", :headers=>{"Content-Type"=>"application/json", "tenantid"=>"%{[fields][tenant_id]}", "apikey"=>"%{[fields][api_key]}"}}

[bigphil (Philip Nunn)]

Look at the config examples here. Looks like you're calling the field names incorrectly. Also see link here for accessing field data.

[Ramalingam_Chandran]

Thanks Philip. So it always needs to be an metadata field?. I was going throught threads and refered this thread for my configuration

Filebeat fields are not working

Here is my filebeat config: filebeat.prospectors: - type: log paths: - C:/inetpub/app01/logs/IIS/*/*.log fields: app_group: iis app_id: app01 - type: log paths: - C:/inetpub/app02/logs/IIS/*/*.log fields: app_group: iis app_id: app02 - type: log paths: - C:/inetpub/app03/logs/IIS/*/*.log fields: app_group: iis app_id: app03 output: logstash: hosts: ["myserver:7777"] And here is logstash: input { beats { port => 7777 } } filter …

[bigphil (Philip Nunn)]

No. You're not working with metadata fields, just normal fields. Metadata fields are not added to output. You need something like this: [tenant_id] since you're declaring it to go under root.

[Ramalingam_Chandran]

Tried by changing the fields to metadata. Still the same issue

replaced with this in the filter section

mutate
{
add_field => ["[@metadata][lmmtenant]", "%{[fields][tenant_id]"]
add_field => ["[@metadata][lmmapikey]", "%{[fields][api_key]"]
}

Event output:
:headers=>{"Content-Type"=>"application/json", "tenantid"=>"%{[@metadata][lmmtenant]}", "apikey"=>"%{[@metadata][lmmapikey]}"}

[Ramalingam_Chandran]

I removed the under root option and did the following configuration

filter
{
mutate
{
add_field => ["tenant_id", "%{[fields][log_tenant_id]}"]
add_field => ["apikey", "%{[fields][log_api_key]}"]
}

}

output
{
http
{
url => "xxxxxxxxxxxx"
http_method => "post"
format => "json_batch"
headers => ["Content-Type", "application/json"]
headers => ["tenantid", "%{tenant_id}"]
headers => ["apikey", "%{apikey}"]
}
}

I can see the filebeat fields are getting assinged in the filter section. But not in the output section headers.

2018-12-04T19:27:51,397][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"tenant_id", "value"=>["%{[fields][log_tenant_id]}"]}

[2018-12-04T19:27:51,398][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"apikey", "value"=>["%{[fields][log_api_key]}"]}

[2018-12-04T19:27:51,409][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"tenant_id", "value"=>["%{[fields][log_tenant_id]}"]}

[2018-12-04T19:27:51,409][DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"apikey", "value"=>["%{[fields][log_api_key]}"]}

[2018-12-04T19:27:51,479][DEBUG][logstash.pipeline ] output received {"event"=>{"host"=>{"name"=>"xxxxxxx"}, "fields"=>{"log_tenant_id"=>"xxxxxxx", "log_api_key"=>"xxxxxxxx", "type"=>"access_log"}, "@timestamp"=>2018-12-04T19:27:49.088Z, "tenant_id"=>"xxxxxxx", "offset"=>0, "input"=>{"type"=>"log"}, "apikey"=>"xxxxxxxxxx", "beat"=>{"hostname"=>"xxxxxxx", "version"=>"6.4.1", "name"=>"xxxxxx"}, "message"=>"127.0.0.1 - - [09/Oct/2018:14:28:30 +0000] "GET /1/config HTTP/1.1" 504 1 [504] [10.255.255.0:80] [0] [60.060]", "source"=>"/var/log/nginx/access.log", "prospector"=>{"type"=>"log"}, "tags"=>["beats_input_codec_plain_applied"], "@version"=>"1"}}

[Ramalingam_Chandran]

Hi Philip

Thanks a lot for your help and suggestions. The issue is with logstash http output format. After changing the format from "json_batch" to "json" , I am able to get the field value in the output.