Kafka output logs everything with WARN level

imehl · September 29, 2017, 12:51pm

Hi,

the kafka output plugin of beats logs all messages with the "WARN" level.

Example of "INFO" messages:

2017/09/29 12:44:40.975292 log.go:16: WARN kafka message: Initializing new client
2017/09/29 12:44:40.975362 log.go:12: WARN client/metadata fetching metadata for all topics from broker broker01:9092
2017/09/29 12:44:41.057287 log.go:12: WARN Connected to broker at broker01:9092 (unregistered)
2017/09/29 12:44:41.066633 log.go:12: WARN client/brokers registered new broker #2 at broker02:9092
2017/09/29 12:44:41.066656 log.go:12: WARN client/brokers registered new broker #5 at broker05:9092
2017/09/29 12:44:41.066666 log.go:12: WARN client/brokers registered new broker #4 at broker04:9092
2017/09/29 12:44:41.066675 log.go:12: WARN client/brokers registered new broker #7 at broker07:9092
2017/09/29 12:44:41.066683 log.go:12: WARN client/brokers registered new broker #1 at broker01:9092
2017/09/29 12:44:41.066691 log.go:12: WARN client/brokers registered new broker #3 at broker03:9092
2017/09/29 12:44:41.066700 log.go:12: WARN client/brokers registered new broker #6 at broker06:9092
2017/09/29 12:44:41.066753 log.go:16: WARN kafka message: Successfully initialized new client
2017/09/29 12:44:41.066998 log.go:12: WARN producer/broker/2 starting up
2017/09/29 12:44:41.067019 log.go:12: WARN producer/broker/2 state change to [open] on topic1/0
2017/09/29 12:44:41.067096 log.go:12: WARN producer/broker/6 starting up
2017/09/29 12:44:41.067109 log.go:12: WARN producer/broker/6 state change to [open] on topic1/4
2017/09/29 12:44:41.067267 log.go:12: WARN producer/broker/3 starting up
2017/09/29 12:44:41.067284 log.go:12: WARN producer/broker/3 state change to [open] on topic1/1
2017/09/29 12:44:41.067380 log.go:12: WARN producer/broker/4 starting up
2017/09/29 12:44:41.067397 log.go:12: WARN producer/broker/4 state change to [open] on topic1/2
2017/09/29 12:44:41.067619 log.go:12: WARN producer/broker/5 starting up
2017/09/29 12:44:41.067638 log.go:12: WARN producer/broker/5 state change to [open] on topic1/3
2017/09/29 12:44:41.067809 log.go:12: WARN producer/broker/7 starting up
2017/09/29 12:44:41.067824 log.go:12: WARN producer/broker/7 state change to [open] on topic1/5
2017/09/29 12:44:41.067875 log.go:12: WARN producer/broker/1 starting up
2017/09/29 12:44:41.067887 log.go:12: WARN producer/broker/1 state change to [open] on topic1/6
2017/09/29 12:44:41.108484 log.go:12: WARN Connected to broker at broker05:9092 (registered as #5)
2017/09/29 12:44:41.112018 log.go:12: WARN Connected to broker at broker02:9092 (registered as #2)
2017/09/29 12:44:41.115963 log.go:12: WARN Connected to broker at broker06:9092 (registered as #6)
2017/09/29 12:44:41.117159 log.go:12: WARN Connected to broker at broker04:9092 (registered as #4)
2017/09/29 12:44:41.117633 log.go:12: WARN Connected to broker at broker07:9092 (registered as #7)
2017/09/29 12:44:41.123645 log.go:12: WARN Connected to broker at broker03:9092 (registered as #3)
2017/09/29 12:44:41.126504 log.go:12: WARN Connected to broker at broker01:9092 (registered as #1)

Which results in wrong insights in later log data analytics - is this a bug?

steffens · October 4, 2017, 11:24am

Unfortunately it's a limitation of the kafka client library used by beats.

imehl · October 4, 2017, 9:29pm

I have looked at the sarama code... It passes error objects (ErrLeaderNotAvailable, ErrTopicAuthorizationFailed,...) for critical errors and strings for info and warnings.

So we could check the input interface type in:

github.com

elastic/beats/blob/master/libbeat/outputs/kafka/log.go

package kafka

import (
	"github.com/Shopify/sarama"

	"github.com/elastic/beats/libbeat/logp"
)

type kafkaLogger struct{}

func (kl kafkaLogger) Print(v ...interface{}) {
	kl.Log("kafka message: %v", v)
}

func (kl kafkaLogger) Printf(format string, v ...interface{}) {
	kl.Log(format, v)
}

func (kl kafkaLogger) Println(v ...interface{}) {
	kl.Log("kafka message: %v", v...)

This file has been truncated. show original

And here I would suggest to set the level to INFO for input strings, and to WARN/ERR for error objects.

I can make a PR if that makes sense

steffens · October 5, 2017, 9:45am

Sounds good to me. A PR with this change is very welcome.

system · October 20, 2017, 12:52pm

This topic was automatically closed after 21 days. New replies are no longer allowed.