i have kafka and packetbeat in same machine of ubuntu distribution. i can get the network statistics of 80 and 53 port from packetbeat but i would like to know is there any way we can get the kafka traffic.
i have seen the option of the process in packetbeat.yml which makes me to try with kafka.
here is my configuration:
packetbeat.yml:
interfaces:
device: any
protocols:
dns:
ports: [53]
include_authorities: true
include_additionals: true
http:
ports: [80, 8080, 8000, 5000, 8002]
procs:
enabled: true - process: kafk-1 cmdline_grep: QuorumPeerMain
output:
logstash: hosts: ["fqdn:5048"]
shipper:
tags: ["kafka-01"] ignore_outgoing: false
logging:
to_files: true files:
path: /var/log/packbeat
name: packbeat.log
rotateeverybytes: 10485760 # = 10MB
i have started the zookeeper service from the port 2181 and kafka server with 9093.
process name i referred from the path /proc/pid/cmdline.
zookeeper has cmdline:
/usr/bin/java-cp/etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr/share/java/slf4j-log4j12.jar:/usr/share/java/zookeeper.jar-Dcom.sun.management.jmxremote-Dcom.sun.management.jmxremote.local.only=false-Dzookeeper.log.dir=/var/log/zookeeper-Dzookeeper.root.logger=INFO,ROLLINGFILEorg.apache.zookeeper.server.quorum.QuorumPeerMain/etc/zookeeper/conf/zoo.cfg
kafka server has cmdline:
java-Xmx1G-Xms1G-server-XX:+UseG1GC-XX:MaxGCPauseMillis=20-XX:InitiatingHeapOccupancyPercent=35-XX:+DisableExplicitGC-Djava.awt.headless=true-Xloggc:/home/kafka/kafka/bin/../logs/kafkaServer-gc.log-verbose:gc-XX:+PrintGCDetails-XX:+PrintGCDateStamps-XX:+PrintGCTimeStamps-Dcom.sun.management.jmxremote-Djava.rmi.server.hostname=(publicaddr)-Dcom.sun.management.jmxremote.authenticate=false-Dcom.sun.management.jmxremote.ssl=false-Dkafka.logs.dir=/home/kafka/kafka/bin/../logs-Dlog4j.configuration=file:bin/../config/log4j.properties-cp:/home/kafka/kafka/bin/../libs/*kafka.Kafkaconfig/server-1.properties
i have mentioned the below for the zookeeper in packetbeat.yml:
cmdline_grep: QuorumPeerMain
question:
- after the above packetbeat configuration, i have started the packetbeat service in debug mode, i dont see network socket from the process name into elasticsearch nor i am receiving in logstash.
2.i don't have idea how to mention cmdline_grep process name for kafka server from the /proc/pid/cmdline. i also tried mentioning the process name from /proc/pid/comm but it was given "java". can we mention java as process name in packetbeat.yml?
your help would be much appreciated.
thanks in advance.