Can we get network metrics of kafka through packetbeat?

Elastic Stack Beats
1

i have kafka and packetbeat in same machine of ubuntu distribution. i can get the network statistics of 80 and 53 port from packetbeat but i would like to know is there any way we can get the kafka traffic.
i have seen the option of the process in packetbeat.yml which makes me to try with kafka.

here is my configuration:
packetbeat.yml:

interfaces:
device: any
protocols:
dns:
ports: [53]
include_authorities: true
include_additionals: true

http:

 ports: [80, 8080, 8000, 5000, 8002]

procs:

enabled: true
      - process: kafk-1
       cmdline_grep: QuorumPeerMain

output:

  logstash:
              hosts: ["fqdn:5048"]

shipper:

 tags: ["kafka-01"]
 ignore_outgoing: false

logging:

to_files: true
 files:

path: /var/log/packbeat
name: packbeat.log
rotateeverybytes: 10485760 # = 10MB

i have started the zookeeper service from the port 2181 and kafka server with 9093.
process name i referred from the path /proc/pid/cmdline.

zookeeper has cmdline:
/usr/bin/java-cp/etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar-Dcom.sun.management.jmxremote-Dcom.sun.management.jmxremote.local.only=false-Dzookeeper.log.dir=/var/log/zookeeper-Dzookeeper.root.logger=INFO,ROLLINGFILEorg.apache.zookeeper.server.quorum.QuorumPeerMain/etc/zookeeper/conf/zoo.cfg

kafka server has cmdline:
java-Xmx1G-Xms1G-server-XX:+UseG1GC-XX:MaxGCPauseMillis=20-XX:InitiatingHeapOccupancyPercent=35-XX:+DisableExplicitGC-Djava.awt.headless=true-Xloggc:/home/kafka/kafka/bin/../logs/kafkaServer-gc.log-verbose:gc-XX:+PrintGCDetails-XX:+PrintGCDateStamps-XX:+PrintGCTimeStamps-Dcom.sun.management.jmxremote-Djava.rmi.server.hostname=(publicaddr)-Dcom.sun.management.jmxremote.authenticate=false-Dcom.sun.management.jmxremote.ssl=false-Dkafka.logs.dir=/home/kafka/kafka/bin/../logs-Dlog4j.configuration=file:bin/../config/log4j.properties-cp:/home/kafka/kafka/bin/../libs/*kafka.Kafkaconfig/server-1.properties

i have mentioned the below for the zookeeper in packetbeat.yml:
cmdline_grep: QuorumPeerMain

question:

  1. after the above packetbeat configuration, i have started the packetbeat service in debug mode, i dont see network socket from the process name into elasticsearch nor i am receiving in logstash.

2.i don't have idea how to mention cmdline_grep process name for kafka server from the /proc/pid/cmdline. i also tried mentioning the process name from /proc/pid/comm but it was given "java". can we mention java as process name in packetbeat.yml?

your help would be much appreciated.

thanks in advance.

2

packetbeat is a passive protocol analyzer, reassembling tcp streams from network packets. No kafka protocol analyzer is implemented yet (but shouldn't be too hard). Alternatively one can use metricbeat kafka and zookeeper modules to pull some stats right from kafka.

  1. after the above packetbeat configuration, i have started the packetbeat service in debug mode, i dont see network socket from the process name into elasticsearch nor i am receiving in logstash.

Anything in logs? Logstash reachable? Please format logs, files and command lines using 3 backticks '`' to make them readable and keep format. I can not tell if your config file is correct or not without formatting.

2.i don't have idea how to mention cmdline_grep process name for kafka server from the /proc/pid/cmdline. i also tried mentioning the process name from /proc/pid/comm but it was given "java". can we mention java as process name in packetbeat.yml?

I don't quite understand what kind of problem you try to solve here.

3

Kafka has a lot of JMX metrics that might solve some of your requirements.

4

There is also a communitybeat collecting kafka jmx stats: https://github.com/gingerwizard/kafkabeat