How to set Log4J level on individual Logstash plugin?

Elastic Stack Logstash
1

Background: I've been working extensively with the Logstash Netflow codec the past few days. Although it's been working fairly well, it's been filling my logs with a ton of messages like:

...
[2017-02-08T00:52:58,724][WARN ][logstash.codecs.netflow  ] Ignoring Netflow version v10
[2017-02-08T00:52:58,724][WARN ][logstash.codecs.netflow  ] Ignoring Netflow version v10
[2017-02-08T00:52:58,724][WARN ][logstash.codecs.netflow  ] Ignoring Netflow version v10
...

(I am ignoring v10 on purpose, and cannot turn off those messages)

These logs are becoming a problem due to:

  • The excessive amount of disk space used by the log files
  • The drain on computer resources as it writes the excessive amount of warning messages to disk.

What I'd ultimately like to do is instruct Log4J to only log the Netflow codec plugin's messages when they are ERROR+. But still see INFO+ messages from other plugins/logstash core.

Is there a way to set configuration for a specific plugin's logger? At the moment I have a pretty standard log4j2.properties (I assume taken from some arbitrary example):

appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %-.10000m%n

appender.json_rolling.type = RollingFile
appender.json_rolling.name = json_rolling
appender.json_rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
appender.json_rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}.log
appender.json_rolling.policies.type = Policies
appender.json_rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.json_rolling.policies.time.interval = 1
appender.json_rolling.policies.time.modulate = true
appender.json_rolling.layout.type = JSONLayout
appender.json_rolling.layout.compact = true
appender.json_rolling.layout.eventEol = true

rootLogger.level = ${sys:ls.log.level}
rootLogger.appenderRef.rolling.ref = ${sys:ls.log.format}_rolling
2

Got it!

First, a co-worker pointed out that Logstash 5's HTTP API has some documentation around getting or setting log levels: https://www.elastic.co/guide/en/logstash/current/logging.html

So after a bit of RTFM, looks like we can dynamically set log levels on a plugin-level basis! But it's an ephemeral setting - I needed something more permanent.

I found adding the following to the bottom of my log4j2.properties does the trick:

logger.netflow.name = logstash.codecs.netflow
logger.netflow.level = ERROR

This plugin is now blissfully silent.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.