Logstash kv filter include_keys

Elastic Stack Logstash
1

Hello Logstash experts,
Is it ok to have multiple include_keys in my kv filter config? I want to avoid parsing all KVs since it is adding a lot of junk KV pairs to my list of fields, so I am whitelisting the keys I am interested in. The whitelist has grown over a period of time. Configtest seems to allow this, and logstash came up fine but I am not seeing the keys I expect, so was wondering if this is wrong:

  kv {
     source => "message"
     include_keys => ["received_from", "count", "mean_rate", "m1", "m5", "m15", "min", "max", "mean", "stddev", "median", "p75", "p95", "p98", "p99", "p999"]
     include_keys => ["mean_rate", "rate_unit", "duration_unit", "type", "name", "appName", "contextId", "gateway", "route", "deviceId", "messageType", "errStack"]
     include_keys => ["errMessage", "messageDir", "mqttTopic", "topic", "allowAnonymous", "stats.allowedWriteCounter", "stats.deniedWriteCounter", "stats.allowedReadCounter"]
     trim => "<>\[\],"
     trimkey => "<>\[\],"
    }
2

The docs don't really state you cannot, but does it work?

3

Update, it works fine (which is good, no need for long lines in config). The issue was something else in my config which caused my test to skip the kv filter altogether.

Thanks

1 Like