# Data disparity between Kibana and database **URL:** \ **Category:** Beats\ **Tags:** packetbeat\ **Created:** [January 27, 2016, 12:25pm UTC](https://discuss.elastic.co/t/data-disparity-between-kibana-and-database/40227 "2016-01-27T12:25:01Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![karlisson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karlisson/32/7404_2.png) [@karlisson](https://discuss.elastic.co/u/karlisson)\ **Post date:** [January 27, 2016, 12:25pm UTC](https://discuss.elastic.co/t/data-disparity-between-kibana-and-database/40227/1 "2016-01-27T12:25:01Z")
Hello! I'm using packetbeat with pf\_ring to gather data from Mysql INSERT's and index it in ES, showing on Kibana. pf\_ring works on a machine which has 20k packets per second and in total sums 280GB per day. It is an ElasticSearch cluster with 3 nodes m3.3xlarge. The problem is: there's a difference between the data shown in Kibana and the data in MySQL. Below, you can see this difference (Index/Database) in one day, per hour (0h-23h): ![](https://us1.discourse-cdn.com/elastic/original/2X/7/70468ccd8fa205dec9efecc8ddb0ed54cc7407b5.png) What possibly can be causing this? ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [January 27, 2016, 7:41pm UTC](https://discuss.elastic.co/t/data-disparity-between-kibana-and-database/40227/2 "2016-01-27T19:41:06Z")
Not sure I understand this, what are the numbers in the diff column? Number of rows/documents? ---
**Author:** ![slourenco](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slourenco/32/5780_2.png) [@slourenco](https://discuss.elastic.co/u/slourenco)\ **Post date:** [January 28, 2016, 10:33am UTC](https://discuss.elastic.co/t/data-disparity-between-kibana-and-database/40227/3 "2016-01-28T10:33:19Z")
Hi @tudor ! In fact the issue here is that we have many servers receiving data from packetbeat strait to this 3 elastic server cluster, on top of a loadbalancer for saving data. In this box, we have something like 20k tcp packets/s. On the others we have something obout 10K packets/s in a total of 42 servers. This totalizes about 300gb of data and 260M documents per day. Now the issue we have is that we are missing information from the servers; those numbers just indicates the amount missed in da single day only for mysql packets. We don't know if the problem is on packetbeat dropping data, the elasticsearch time outing the saves or even something about the pf\_ring integration; since pf\_ring should provide 50k packets/s i think the problem is not it. - How can I debug the integration of pf\_ring and packetbeat? - How can I debug the packetbeat parsing ? - How can I debug the data from packetbeat to elasticsearch? - and finally but not least, how can I debug elasticsearch for dropping saves or indexing ? We a planing to build a HUGE server with 5 nodes and 100TB of data and 100 billion documents. many thanks. ps. we are using packetbeat 1.0.1 compiled from source for pf\_ring support. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:56pm UTC](https://discuss.elastic.co/t/data-disparity-between-kibana-and-database/40227/4 "2017-07-05T21:56:14Z")