Kafka output logs everything with WARN level


(Michael) #1

Hi,

the kafka output plugin of beats logs all messages with the "WARN" level.

Example of "INFO" messages:

2017/09/29 12:44:40.975292 log.go:16: WARN kafka message: Initializing new client
2017/09/29 12:44:40.975362 log.go:12: WARN client/metadata fetching metadata for all topics from broker broker01:9092
2017/09/29 12:44:41.057287 log.go:12: WARN Connected to broker at broker01:9092 (unregistered)
2017/09/29 12:44:41.066633 log.go:12: WARN client/brokers registered new broker #2 at broker02:9092
2017/09/29 12:44:41.066656 log.go:12: WARN client/brokers registered new broker #5 at broker05:9092
2017/09/29 12:44:41.066666 log.go:12: WARN client/brokers registered new broker #4 at broker04:9092
2017/09/29 12:44:41.066675 log.go:12: WARN client/brokers registered new broker #7 at broker07:9092
2017/09/29 12:44:41.066683 log.go:12: WARN client/brokers registered new broker #1 at broker01:9092
2017/09/29 12:44:41.066691 log.go:12: WARN client/brokers registered new broker #3 at broker03:9092
2017/09/29 12:44:41.066700 log.go:12: WARN client/brokers registered new broker #6 at broker06:9092
2017/09/29 12:44:41.066753 log.go:16: WARN kafka message: Successfully initialized new client
2017/09/29 12:44:41.066998 log.go:12: WARN producer/broker/2 starting up
2017/09/29 12:44:41.067019 log.go:12: WARN producer/broker/2 state change to [open] on topic1/0
2017/09/29 12:44:41.067096 log.go:12: WARN producer/broker/6 starting up
2017/09/29 12:44:41.067109 log.go:12: WARN producer/broker/6 state change to [open] on topic1/4
2017/09/29 12:44:41.067267 log.go:12: WARN producer/broker/3 starting up
2017/09/29 12:44:41.067284 log.go:12: WARN producer/broker/3 state change to [open] on topic1/1
2017/09/29 12:44:41.067380 log.go:12: WARN producer/broker/4 starting up
2017/09/29 12:44:41.067397 log.go:12: WARN producer/broker/4 state change to [open] on topic1/2
2017/09/29 12:44:41.067619 log.go:12: WARN producer/broker/5 starting up
2017/09/29 12:44:41.067638 log.go:12: WARN producer/broker/5 state change to [open] on topic1/3
2017/09/29 12:44:41.067809 log.go:12: WARN producer/broker/7 starting up
2017/09/29 12:44:41.067824 log.go:12: WARN producer/broker/7 state change to [open] on topic1/5
2017/09/29 12:44:41.067875 log.go:12: WARN producer/broker/1 starting up
2017/09/29 12:44:41.067887 log.go:12: WARN producer/broker/1 state change to [open] on topic1/6
2017/09/29 12:44:41.108484 log.go:12: WARN Connected to broker at broker05:9092 (registered as #5)
2017/09/29 12:44:41.112018 log.go:12: WARN Connected to broker at broker02:9092 (registered as #2)
2017/09/29 12:44:41.115963 log.go:12: WARN Connected to broker at broker06:9092 (registered as #6)
2017/09/29 12:44:41.117159 log.go:12: WARN Connected to broker at broker04:9092 (registered as #4)
2017/09/29 12:44:41.117633 log.go:12: WARN Connected to broker at broker07:9092 (registered as #7)
2017/09/29 12:44:41.123645 log.go:12: WARN Connected to broker at broker03:9092 (registered as #3)
2017/09/29 12:44:41.126504 log.go:12: WARN Connected to broker at broker01:9092 (registered as #1)

Which results in wrong insights in later log data analytics - is this a bug?


(Steffen Siering) #2

Unfortunately it's a limitation of the kafka client library used by beats.


(Michael) #3

I have looked at the sarama code... It passes error objects (ErrLeaderNotAvailable, ErrTopicAuthorizationFailed,...) for critical errors and strings for info and warnings.

So we could check the input interface type in:

And here I would suggest to set the level to INFO for input strings, and to WARN/ERR for error objects.

I can make a PR if that makes sense


(Steffen Siering) #4

Sounds good to me. A PR with this change is very welcome.


(system) #5

This topic was automatically closed after 21 days. New replies are no longer allowed.