i am collecting the metrics from kafka through logstash and storing it in elastic search.
basically, i can able to collect the server metrics from kafka but metrics of producer and consumer is not working.
below are my kafka configuration:
producer.properties:
metadata.broker.list=publicipaddr:9093,publicipaddr:9094
producer.type=sync
serializer.class=kafka.serializer.DefaultEncoder
client.id=logstash
consumer.properties:
zookeeper.connect=privateipaddr:2181
group.id=logst-test
client.id=logstash
auto.offset.reset=largest
i could able to send and receive messages by executing the producer and consumer like below:
bin/kafka-console-producer.sh --broker-list publicipaddr:9093,publicipaddr:9094 --sync --topic test1
bin/kafka-console-consumer.sh --zookeeper privateipaddr:2181 --topic test1 --from-beginning --consumer.config config/consumer.properties
my client id and group id is listed while executing:
bin/kafka-consumer-groups.sh --zookeeper privateipaddr:2181 --list
logstash
logst-test
i have specified client id and group id different since i dont know how this works while listening to logstash. so, i have given different name to be listed in groups.
below is my logstash metrics which i have defined in /etc/logstash/jmx folder:
{
"host" : "publicipaddr",
"port" : 9997,
"alias" : "kafka_1_jvm",
"queries" : [
{
"object_name" : "kafka.producer:type=producer-metrics,clientId=${client_id}",
"object_alias" : "kafka_producer_4"
},{
"object_name" : "kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash,broker=0",
"object_alias" : "kafka_producer_2"
}
]
}
i have tried with different possibilities of clientId value like:
1.clientId=${client_id}
2.clientId=logstash
3.clientId=logst-test
4.clientId=([-.\w]+)
whenever i run the logstash:
bin/logstash agent -f /etc/logstash/conf.d/jmx_kafka.conf --debug --verbose
i get the same error message, i dont see metrics coming from kafka:
(1,2,3).No jmx object found for kafka.producer:type=producer-metrics,client-id=${client_id} {:level=>:warn}
4.unrecognised token \w
i am badly struck. please help me
appreciated.