Packetbeat debugging - no file output

tiburblium · May 29, 2017, 11:00pm

I am trying to debug my standalone installation of packetbeat, which I've configured to output to a file. I am not getting any file output and I'm not clear on how to get more information on what the problem would be. Appreciate any tips or advice

This is my output when I run a config test:

root@logshipper:/tmp/packetbeat# /usr/bin/packetbeat.sh -configtest
2017/05/29 22:31:19.211285 beat.go:285: INFO Home path: [/usr/share/packetbeat] Config path: [/etc/packetbeat] Data path: [/var/lib/packetbeat] Logs path: [/var/log/packetbeat]
2017/05/29 22:31:19.211395 beat.go:186: INFO Setup Beat: packetbeat; Version: 5.4.0
2017/05/29 22:31:19.211443 processor.go:44: DBG  Processors: 
2017/05/29 22:31:19.211471 beat.go:192: DBG  Initializing output plugins
2017/05/29 22:31:19.211577 file.go:54: INFO File output path set to: /tmp/yeah
2017/05/29 22:31:19.211630 file.go:55: INFO File output base filename set to: packetbeat
2017/05/29 22:31:19.211668 file.go:58: INFO Rotate every bytes set to: 10240000
2017/05/29 22:31:19.211716 file.go:62: INFO Number of files set to: 7
2017/05/29 22:31:19.211750 outputs.go:108: INFO Activated file as output plugin.
2017/05/29 22:31:19.211776 publish.go:238: DBG  Create output worker
2017/05/29 22:31:19.211843 publish.go:280: DBG  No output is defined to store the topology. The server fields might not be filled.
2017/05/29 22:31:19.211887 publish.go:295: INFO Publisher name: logshipper
2017/05/29 22:31:19.212014 async.go:63: INFO Flush Interval set to: -1s
2017/05/29 22:31:19.212047 async.go:64: INFO Max Bulk Size set to: -1
2017/05/29 22:31:19.212084 procs.go:79: INFO Process matching disabled
2017/05/29 22:31:19.212177 packetbeat.go:109: DBG  Initializing protocol plugins
2017/05/29 22:31:19.212210 protos.go:89: INFO registered protocol plugin: cassandra
2017/05/29 22:31:19.212234 protos.go:89: INFO registered protocol plugin: dns
2017/05/29 22:31:19.212260 protos.go:89: INFO registered protocol plugin: http
2017/05/29 22:31:19.212282 protos.go:89: INFO registered protocol plugin: nfs
2017/05/29 22:31:19.212304 protos.go:89: INFO registered protocol plugin: pgsql
2017/05/29 22:31:19.212326 protos.go:89: INFO registered protocol plugin: redis
2017/05/29 22:31:19.212348 protos.go:89: INFO registered protocol plugin: thrift
2017/05/29 22:31:19.212369 protos.go:89: INFO registered protocol plugin: amqp
2017/05/29 22:31:19.212390 protos.go:89: INFO registered protocol plugin: mongodb
2017/05/29 22:31:19.212411 protos.go:89: INFO registered protocol plugin: mysql
2017/05/29 22:31:19.212433 protos.go:89: INFO registered protocol plugin: memcache
2017/05/29 22:31:19.212454 packetbeat.go:115: DBG  Initializing sniffer
2017/05/29 22:31:19.212479 sniffer.go:270: DBG  BPF filter: ''
2017/05/29 22:31:19.212503 sniffer.go:156: DBG  Sniffer type: pcap device: any
2017/05/29 22:31:19.213486 tcp.go:307: DBG  tcp%!(EXTRA string=Port map: %v, map[uint16]protos.Protocol=map[])
2017/05/29 22:31:19.213530 udp.go:94: DBG  Port map: map[]
2017/05/29 22:31:19.213567 decoder.go:98: DBG  Layer type: Linux SLL
Config OK
root@logshipper:/tmp/packetbeat#

Log file after running packetbeat:

root@logshipper:/tmp/packetbeat# cat /var/log/packetbeat/packetbeat
2017-05-29T22:17:16Z INFO Home path: [/usr/share/packetbeat] Config path: [/etc/packetbeat] Data path: [/var/lib/packetbeat] Logs path: [/var/log/packetbeat]
2017-05-29T22:17:16Z INFO Setup Beat: packetbeat; Version: 5.4.0
2017-05-29T22:17:16Z INFO File output path set to: /tmp/testamundo/
2017-05-29T22:17:16Z INFO File output base filename set to: packetbeat
2017-05-29T22:17:16Z INFO Rotate every bytes set to: 10240000
2017-05-29T22:17:16Z INFO Number of files set to: 7
2017-05-29T22:17:16Z INFO Activated file as output plugin.
2017-05-29T22:17:16Z INFO Publisher name: logshipper
2017-05-29T22:17:16Z INFO Flush Interval set to: -1s
2017-05-29T22:17:16Z INFO Max Bulk Size set to: -1
2017-05-29T22:17:16Z INFO Process matching disabled
2017-05-29T22:17:16Z INFO registered protocol plugin: http
2017-05-29T22:17:16Z INFO registered protocol plugin: memcache
2017-05-29T22:17:16Z INFO registered protocol plugin: mongodb
2017-05-29T22:17:16Z INFO registered protocol plugin: pgsql
2017-05-29T22:17:16Z INFO registered protocol plugin: redis
2017-05-29T22:17:16Z INFO registered protocol plugin: thrift
2017-05-29T22:17:16Z INFO registered protocol plugin: amqp
2017-05-29T22:17:16Z INFO registered protocol plugin: cassandra
2017-05-29T22:17:16Z INFO registered protocol plugin: dns
2017-05-29T22:17:16Z INFO registered protocol plugin: mysql
2017-05-29T22:17:16Z INFO registered protocol plugin: nfs
2017-05-29T22:17:16Z INFO packetbeat start running.
2017-05-29T22:17:16Z INFO Metrics logging every 30s
2017-05-29T22:17:46Z INFO No non-zero metrics in the last 30s
2017-05-29T22:18:16Z INFO No non-zero metrics in the last 30s
2017-05-29T22:18:46Z INFO No non-zero metrics in the last 30s
2017-05-29T22:19:16Z INFO No non-zero metrics in the last 30s
2017-05-29T22:19:46Z INFO No non-zero metrics in the last 30s
2017-05-29T22:20:16Z INFO No non-zero metrics in the last 30s
2017-05-29T22:20:46Z INFO No non-zero metrics in the last 30s
2017-05-29T22:21:16Z INFO No non-zero metrics in the last 30s
2017-05-29T22:21:46Z INFO No non-zero metrics in the last 30s
2017-05-29T22:22:16Z INFO No non-zero metrics in the last 30s
2017-05-29T22:22:46Z INFO No non-zero metrics in the last 30s
2017-05-29T22:23:16Z INFO No non-zero metrics in the last 30s
2017-05-29T22:23:27Z INFO Packetbeat send stop signal

tudor · May 30, 2017, 9:19am

Can you show us your configuration file. Also, I recommend running with the -e -d "*" options, which enable full debugging.

tiburblium · May 30, 2017, 3:36pm

Hi Tudor, please see below, let me know if there is any other information I could provide that would help! Thank you:

Here is my packetbeat.yml:

#### sniffer #########################################
interfaces:
  device: any

############################# Protocols #######################################
protocols:
  dns:
    ports: [53]

    include_authorities: true
    include_additionals: true

  http:
    ports: [80, 8080, 8081, 5000, 8002]

  memcache:
    ports: [11211]

  mysql:
    ports: [3306]

  pgsql:
    ports: [5432]

  redis:
    ports: [6379]

  thrift:
    ports: [9090]

  mongodb:
    ports: [27017]

############################# Output ##########################################
output:
  file:
    path: "/tmp/yeah"
    filename: packetbeat
    rotate_every_kb: 10000
    number_of_files: 7

#output:
#  logstash:
#    hosts: ["elk.zugtastic.com:5044"]

#ssl:
#   certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

############################# Logging #########################################
logging:

  files:
    rotateeverybytes: 10485760 # = 10MB

I installed packetbeat via the debian repository, it added a runner script here that I modified to include the debug output:

root@ip-172-31-0-40:/home/ubuntu# cat /usr/bin/packetbeat.sh 
#!/bin/bash

# Script to run packetbeat in foreground with the same path settings that
# the init script / systemd unit file would do.

/usr/share/packetbeat/bin/packetbeat -e -d "*" \
  -path.home /usr/share/packetbeat \
  -path.config /etc/packetbeat \
  -path.data /var/lib/packetbeat \
  -path.logs /var/log/packetbeat \
  $@

When I run it, this is a tail of the stdout I see:

2017/05/30 15:32:16.972992 decoder.go:143: DBG  decode packet data
2017/05/30 15:32:16.973005 decoder.go:216: DBG  IPv4 packet
2017/05/30 15:32:16.973024 decoder.go:257: DBG  TCP packet
2017/05/30 15:32:16.973041 sniffer.go:379: DBG  Packet number: 12479
2017/05/30 15:32:16.973059 decoder.go:143: DBG  decode packet data
2017/05/30 15:32:16.973076 decoder.go:216: DBG  IPv4 packet
2017/05/30 15:32:16.973091 decoder.go:257: DBG  TCP packet
2017/05/30 15:32:16.973111 sniffer.go:379: DBG  Packet number: 12480
2017/05/30 15:32:16.973127 decoder.go:143: DBG  decode packet data
2017/05/30 15:32:16.973142 decoder.go:216: DBG  IPv4 packet
2017/05/30 15:32:16.973158 decoder.go:257: DBG  TCP packet
^C2017/05/30 15:32:16.997505 service.go:33: DBG  Received sigterm/sigint, stopping
2017/05/30 15:32:16.997565 packetbeat.go:184: INFO Packetbeat send stop signal
2017/05/30 15:32:16.997598 sniffer.go:379: DBG  Packet number: 12481
2017/05/30 15:32:16.997609 decoder.go:143: DBG  decode packet data
2017/05/30 15:32:16.997619 decoder.go:216: DBG  IPv4 packet
2017/05/30 15:32:16.997629 decoder.go:257: DBG  TCP packet
2017/05/30 15:32:16.997639 sniffer.go:384: INFO Input finish. Processed 12481 packets. Have a nice day!
2017/05/30 15:32:16.997749 metrics.go:51: INFO Total non-zero values: 
2017/05/30 15:32:16.997777 metrics.go:52: INFO Uptime: 3.013561625s
2017/05/30 15:32:16.997796 beat.go:225: INFO packetbeat stopped.

Nothing is in the file output directory:

root@ip-172-31-0-40:/home/ubuntu# ls -rlath /tmp/yeah/
total 8.0K
drwxr-x--- 2 root root 4.0K May 29 22:48 .
drwxrwxrwt 9 root root 4.0K May 30 15:28 ..

tiburblium · May 30, 2017, 4:13pm

Success! It is working now, I guess I had an issue with my .yml config for file output, I changed it to be output.file:

output.file:
    path: "/tmp/yeah"
    filename: packetbeat
    rotate_every_kb: 10000
    number_of_files: 7

tudor · May 31, 2017, 8:26am

Ok, note that Packetbeat needs to receive both the request and the replies in order to write a transaction, so you have to leave it running a little bit.

system · June 28, 2017, 8:27am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.