Packetbeat sometimes miss captured packages

Elastic Stack Beats
1

Hi!, I'm using packetbeat to collect data about our billing, monitoring the numbers of inserts in a mysql database table, whatever sometimes it have differences between the numbers returned from elasticsearch and the table inserted lines. I've checked out the cpu, memory, network and logs but it is all ok on the period that it lose the packages. Ex: "February 14th 2016" 6pm to 7pm are 7395 elasticsearch lines versus 11051 table lines.
I'm sure that the only entry of lines on the table came from this channel, and don't have another source inserting on table.
It happens sometimes because I've compared other values of elastic and mysql, and it was correct.

Ex: Another hours
February 15th 2016, 6pm to 7pm are 4225 in elastic versus 4225 table lines.
February 15th 2016, 13pm to 14pm are 13219 in elastic versus 13219 table lines.

I just verified using tcpdump, between the host and the elasticsearch there are no packet lost.

Packetbeat configuration:

interfaces:
  device: eth0
  type: af_packet
  snaplen: 1500
  buffer_size_mb: 500
protocols:
  http:
    ports: [80,443]
    send_response: true
    send_request: true
    send_headers: ["User-Agent"]
    real_ip_header: "X-Forwarded-For"
    include_body_for: ["text/html", "application/json" ,"application/xhtml+xml", "application/xml"]
  mysql:
    ports: [3306]
    max_rows: 100
    max_row_length: 5000
output:
  elasticsearch:
    enabled: true
    worker: 2
    hosts: ["node1","node2","node3"]
    max_retries: 15
    bulk_max_size: 1500
    flush_interval: 20
shipper:
  name: sender1
  tags: ["sender"]
logging:
  to_syslog: false
  to_files: true
  files:
    path: /var/log/
    name: packetbeat.log
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7
  selectors: ["*"]
  level: error

packetbeat version 1.1.1 (amd64)
System: Linux ip-10-1-0-174 4.1.13-19.30.amzn1.x86_64 #1 SMP Fri Dec 11 03:42:10 UTC 2015 x86_64 GNU/Linux

2

Try increasing the log level to info and see if there is anything interesting in the logs. It will be easier to debug the issue once we have some clues. Here are some blind guesses:

  • Any transactions that take more than 10 seconds will be dropped (increase transaction_timeout.
  • Jumbo frames can cause issues (need to increase snaplen)