Hi!, I'm using packetbeat to collect data about our billing, monitoring the numbers of inserts in a mysql database table, whatever sometimes it have differences between the numbers returned from elasticsearch and the table inserted lines. I've checked out the cpu, memory, network and logs but it is all ok on the period that it lose the packages. Ex: "February 14th 2016" 6pm to 7pm are 7395 elasticsearch lines versus 11051 table lines.
I'm sure that the only entry of lines on the table came from this channel, and don't have another source inserting on table.
It happens sometimes because I've compared other values of elastic and mysql, and it was correct.
Ex: Another hours
February 15th 2016, 6pm to 7pm are 4225 in elastic versus 4225 table lines.
February 15th 2016, 13pm to 14pm are 13219 in elastic versus 13219 table lines.
I just verified using tcpdump, between the host and the elasticsearch there are no packet lost.
Packetbeat configuration:
interfaces:
device: eth0
type: af_packet
snaplen: 1500
buffer_size_mb: 500
protocols:
http:
ports: [80,443]
send_response: true
send_request: true
send_headers: ["User-Agent"]
real_ip_header: "X-Forwarded-For"
include_body_for: ["text/html", "application/json" ,"application/xhtml+xml", "application/xml"]
mysql:
ports: [3306]
max_rows: 100
max_row_length: 5000
output:
elasticsearch:
enabled: true
worker: 2
hosts: ["node1","node2","node3"]
max_retries: 15
bulk_max_size: 1500
flush_interval: 20
shipper:
name: sender1
tags: ["sender"]
logging:
to_syslog: false
to_files: true
files:
path: /var/log/
name: packetbeat.log
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
selectors: ["*"]
level: error
packetbeat version 1.1.1 (amd64)
System: Linux ip-10-1-0-174 4.1.13-19.30.amzn1.x86_64 #1 SMP Fri Dec 11 03:42:10 UTC 2015 x86_64 GNU/Linux