Logstash filter to include specific keys only

Elastic Stack Logstash
1

Hello Team,

I have one use case related to logstash filters
I need to parse nested json data with specific fields included and drop rest data

Data format:
{
"_index": "index-name",
"_source": {
"level": "INFO",
"params": {
"language": "id",
"ver": "1"
},
"Id": "some-id",
"logger": "RequestListener",
"httpRequest": {
"requestMethod": "GET",
"userAgent": "external-agent",
},
"message": "xyz-message",
"@timestamp": "2018-08-21T19:18:30.143Z",
"headers": {
"x-forwarded-port": "port"
}
}
}

Say, in above logs below fields should be considered as whitelisted and rest should be dropped

level, httpRequest.requestMethod, message, params.language

I tried using kv filter with include_keys but no luck
Also prune filter does not include "[httpRequest][requestMethod]" as a field

Please let me know what could a miss here?

Thanks

2

Move the nested fields ([httpRequest][requestMethod] and [params][language]) into the top level, then use the prune filter.

1 Like
3

thanks alot @magnusbaeck
implemented it with mutate filter then applied prune white listing on those fields

4

If you get time please check below query

I am seeing drop in log ingestion from cloudwatch plugin, few of the logs are missing at ELK but available in source (cloud watch)

logs are ingested at average rate 6 mil/hour

I couldn't figure out a way to get to root cause of it

thanks.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.