# How to verify filebeat parsed log data count
**URL:** https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055
**Category:** Beats
**Tags:** filebeat
**Created:** [October 3, 2016, 11:23am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055 "2016-10-03T11:23:53Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@sarathymv](https://discuss.elastic.co/u/sarathymv)
#### Post date: [October 3, 2016, 11:23am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/1 "2016-10-03T11:23:53Z")
All,
We are using logstash to parse our application log and index to elastic search. Following is the high level architecture
Webserver with file beat -----\> Logstash ----\> Elastic search ------\> kibana.
We installed file beat (multiline log and we added parser logic) in our 4 web servers and all are sending log data to logstash. In turn, logstash sends the data to elastic search for indexing.
We could see in kibana that all the log data are not indexed. How can we verify that filebeat parsed all the data?
let me know if you need any other information.
Thanks in advance.
---
### Author:  [@tudor](https://discuss.elastic.co/u/tudor)
#### Post date: [October 4, 2016, 9:39am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/2 "2016-10-04T09:39:58Z")
Some options:
- Look in the registry file (location depends on the way you installed, it's `/var/lib/filebeat/registry` on DEB/RPM) and check how far filebeat got into the files.
- Increase logging verbosity in filebeat to info level and check if it writes data
- Increase verbosity of Logstash to check that data reaches LS
- Use the stdout Logstash output to check that the data passes through LS
---
### Author:  [@sarathymv](https://discuss.elastic.co/u/sarathymv)
#### Post date: [October 5, 2016, 2:49pm UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/3 "2016-10-05T14:49:21Z")
Thanks tudor for your suggestions. I verified the registry, file beat gone through the files. But i am not able to check by writing the output to a file as it is running in production. Is there a way to find the count of records which is sent by file beat to logstash? Same way logstash to ELK. If so it is easier to verify. As we see in kibana there are records missing for indexing which means some records are not indexed.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 5, 2016, 7:01pm UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/4 "2016-10-05T19:01:40Z")
Which filebeat version are you using? Can you share your config?
In filebeat 5.0 we write some internal stats which could be read out when enabling -httpprof, but that is not recommended in production. At least in 5.0 you get in the logs every 30s a diff of the change stats which could be helpful in your case.
---
### Author:  [@sarathymv](https://discuss.elastic.co/u/sarathymv)
#### Post date: [October 6, 2016, 12:13pm UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/5 "2016-10-06T12:13:27Z")
Hi ruflin,
We are using Filebeat version 1.2.3
Here is my config file.
Filebeat.yml
```auto
filebeat:
# List of prospectors to fetch data.
prospectors:
# Each - is a prospector. Below are the prospector specific configurations
-
paths:
- /my_app_path/log/file_name_*.log
document_type: applicationlog
input_type: log
multiline:
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
pattern: "^[[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2} [[:word:]]{4}:"
# Defines if the pattern set under pattern should be negated or not. Default is false.
negate: true
match: after
-
# Configuration to use stdin input
#input_type: stdin
paths:
- /apachelog/logs/access_log
document_type: apachelog
registry_file: /var/lib/filebeat/registry
output:
logstash:
# The Logstash hosts
hosts: ["xxx.xxx.xxx.000:5044"]
bulk_max_size: 8
logging:
to_files: true
# To enable logging to files, to_files option has to be set to true
files:
# The directory where the log files will written to.
path: /log_file_path/filebeat
# The name of the files where the logs are written to.
name: prod_filebeat_log
# Configure log file size limit. If limit is reached, log file will be
# automatically rotated
rotateeverybytes: 10485760 # = 10MB
# Number of rotated log files to keep. Oldest files will be deleted first.
keepfiles: 7
level: info
```
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 6, 2016, 12:53pm UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/6 "2016-10-06T12:53:42Z")
Did you have a chance to try out filebeat 5.0? Do you see any errors in the log file? Any chance to share the log file?
Side note which should not have an affect: `bulk_max_size` does not exist in logstash output.
---
### Author:  [@sarathymv](https://discuss.elastic.co/u/sarathymv)
#### Post date: [October 6, 2016, 4:54pm UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/7 "2016-10-06T16:54:32Z")
Thanks ruflin for your reply. I have not tried with filebeat 5.0 and i am not sure whether i can do it. As the prod version is 1.2.3 and if i want to change, it has lot of procedure. Let me try. Log files i will try to get it. Do you want the full log file or few recent data from it?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 7, 2016, 6:55am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/8 "2016-10-07T06:55:18Z")
Full log file would be nice. Is it easier for you to update to 1.3.1?
---
### Author:  [@sarathymv](https://discuss.elastic.co/u/sarathymv)
#### Post date: [October 11, 2016, 8:59am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/9 "2016-10-11T08:59:55Z")
Hi ruflin,
I gave a request to get production log. Meanwhile, If we update filebeat version to 1.3.1, what would be the advantages? Will it solve my current issue?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 11, 2016, 12:20pm UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/10 "2016-10-11T12:20:51Z")
Thanks. The main advantage of 1.3.1 is that the log messages should be improved. But to get all the metrics it requires filebeat 5.0. If it will solve your issue I don't know as I haven't figured out yet what is causing it 🙂
---
### Author:  [@sarathymv](https://discuss.elastic.co/u/sarathymv)
#### Post date: [October 13, 2016, 9:46am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/11 "2016-10-13T09:46:18Z")
Hi ruflin,
I am not able to attach log file as it allows only image format files.
**here is a sample file beat log** :
2016-10-13T09:12:04Z INFO Events sent: 551
2016-10-13T09:12:04Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:11Z INFO Events sent: 534
2016-10-13T09:12:11Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:17Z INFO Events sent: 355
2016-10-13T09:12:17Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:24Z INFO Events sent: 603
2016-10-13T09:12:24Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:32Z INFO Events sent: 559
2016-10-13T09:12:32Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:39Z INFO Events sent: 542
2016-10-13T09:12:39Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:48Z INFO Events sent: 487
2016-10-13T09:12:48Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:50Z INFO Events sent: 304
2016-10-13T09:12:50Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:58Z INFO Events sent: 523
2016-10-13T09:12:58Z INFO Registry file updated. 65 states written.
2016-10-13T09:13:08Z INFO Events sent: 466
2016-10-13T09:13:08Z INFO Registry file updated. 65 states written.
**Sample filebeat registry log:**
/log/production\_2016-10-05.log":{"source":"/log/production\_2016-10-05.log","offset":1446352119,"FileStateOS":{"inode":3145784,"device":51728}},"/log/production\_2016-10-06.log":{"source":"/log/production\_2016-10-06.log","offset":1500169936,"FileStateOS":{"inode":3145785,"device":51728}},"/log/production\_2016-10-07.log":{"source":"/log/production\_2016-10-07.log","offset":1466480344,"FileStateOS":{"inode":3145781,"device":51728}},"/log/production\_2016-10-08.log":{"source":"/log/production\_2016-10-08.log","offset":1137720684,"FileStateOS":{"inode":3145788,"device":51728}},"/log/production\_2016-10-09.log":{"source":"/log/production\_2016-10-09.log","offset":1097745077,"FileStateOS":{"inode":3145787,"device":51728}},"/log/production\_2016-10-10.log":{"source":"/log/production\_2016-10-10.log","offset":1409331639,"FileStateOS":{"inode":3145786,"device":51728}},"/log/production\_2016-10-11.log":{"source":"/log/production\_2016-10-11.log","offset":1333179284,"FileStateOS":{"inode":3145790,"device":51728}},"/log/production\_2016-10-12.log":{"source":"/log/production\_2016-10-12.log","offset":12035204,"FileStateOS":{"inode":3145789,"device":51728}},"/log/production\_2016-10-13.log":{"source":"/log/production\_2016-10-13.log","offset":22612575,"FileStateOS":{"inode":3145803,"device":51728}},"/var/log/apache2/access.log":{"source":"/var/log/apache2/access.log","offset":146782734,"FileStateOS":{"inode":3407877,"device":51728}}}
Please let me know if you need any other details.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 13, 2016, 11:38am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/12 "2016-10-13T11:38:27Z")
Looking at the excerpt of the log above all looks fine. In your initial post, you state that not all events showed up. What was your method to verify that or is it currently only an assumption and that is why you want a method to verify?
---
### Author:  [@sarathymv](https://discuss.elastic.co/u/sarathymv)
#### Post date: [October 13, 2016, 11:46am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/13 "2016-10-13T11:46:40Z")
thanks for the reply. We verify the number records against AWS ELB monitoring. In Monitoring tool we could see number of https success response which is not tallied with the indexed data shown in kibana.
For example, AWS shows there are 2500 http (200) response for a particular time. At the same time if we verify with kibana it shows nearly 2000 http (200) response.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 16, 2016, 1:50pm UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/14 "2016-10-16T13:50:06Z")
@sarathymv I don't know the details of the AWS ELB monitoring, but not 100% sure if this comparison is fully reliable. It could also be that the multiline pattern is not correct. Did you find some specific events which are missing? Is the access\_log file rotated?
---
### Author:  [@sarathymv](https://discuss.elastic.co/u/sarathymv)
#### Post date: [October 16, 2016, 2:08pm UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/15 "2016-10-16T14:08:37Z")
I am not able to find the missing events and do not know the way to find it. Yes access\_log is rotating regularly.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [October 17, 2016, 11:09am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/16 "2016-10-17T11:09:37Z")
I would recommend to also have the rotated names of the `access_log` in your prospector config. It should work as you configured above as it will just keep the file open, but it depends on the algorithm you use for truncation.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 24, 2016, 11:23am UTC](https://discuss.elastic.co/t/how-to-verify-filebeat-parsed-log-data-count/62055/17 "2016-10-24T11:23:54Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.