How to verify filebeat parsed log data count

Elastic Stack Beats
1

All,

We are using logstash to parse our application log and index to elastic search. Following is the high level architecture

Webserver with file beat -----> Logstash ----> Elastic search ------> kibana.

We installed file beat (multiline log and we added parser logic) in our 4 web servers and all are sending log data to logstash. In turn, logstash sends the data to elastic search for indexing.

We could see in kibana that all the log data are not indexed. How can we verify that filebeat parsed all the data?

let me know if you need any other information.

Thanks in advance.

2

Some options:

  • Look in the registry file (location depends on the way you installed, it's /var/lib/filebeat/registry on DEB/RPM) and check how far filebeat got into the files.
  • Increase logging verbosity in filebeat to info level and check if it writes data
  • Increase verbosity of Logstash to check that data reaches LS
  • Use the stdout Logstash output to check that the data passes through LS
3

Thanks tudor for your suggestions. I verified the registry, file beat gone through the files. But i am not able to check by writing the output to a file as it is running in production. Is there a way to find the count of records which is sent by file beat to logstash? Same way logstash to ELK. If so it is easier to verify. As we see in kibana there are records missing for indexing which means some records are not indexed.

4

Which filebeat version are you using? Can you share your config?

In filebeat 5.0 we write some internal stats which could be read out when enabling -httpprof, but that is not recommended in production. At least in 5.0 you get in the logs every 30s a diff of the change stats which could be helpful in your case.

5

Hi ruflin,

We are using Filebeat version 1.2.3

Here is my config file.
Filebeat.yml

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      paths:
        - /my_app_path/log/file_name_*.log
      document_type: applicationlog
      input_type: log
      multiline:

        # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
        pattern: "^[[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2} [[:word:]]{4}:"

        # Defines if the pattern set under pattern should be negated or not. Default is false.
        negate: true
        match: after

    -
      # Configuration to use stdin input
      #input_type: stdin
      paths:
        - /apachelog/logs/access_log
      document_type: apachelog
    
  registry_file: /var/lib/filebeat/registry

output:

 logstash:
    # The Logstash hosts
    hosts: ["xxx.xxx.xxx.000:5044"]
    bulk_max_size: 8
logging:

  to_files: true

  # To enable logging to files, to_files option has to be set to true
  files:
    # The directory where the log files will written to.
    path: /log_file_path/filebeat

    # The name of the files where the logs are written to.
    name: prod_filebeat_log

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    keepfiles: 7

  level: info
6

Did you have a chance to try out filebeat 5.0? Do you see any errors in the log file? Any chance to share the log file?

Side note which should not have an affect: bulk_max_size does not exist in logstash output.

7

Thanks ruflin for your reply. I have not tried with filebeat 5.0 and i am not sure whether i can do it. As the prod version is 1.2.3 and if i want to change, it has lot of procedure. Let me try. Log files i will try to get it. Do you want the full log file or few recent data from it?

8

Full log file would be nice. Is it easier for you to update to 1.3.1?

9

Hi ruflin,

I gave a request to get production log. Meanwhile, If we update filebeat version to 1.3.1, what would be the advantages? Will it solve my current issue?

10

Thanks. The main advantage of 1.3.1 is that the log messages should be improved. But to get all the metrics it requires filebeat 5.0. If it will solve your issue I don't know as I haven't figured out yet what is causing it :slight_smile:

11

Hi ruflin,

I am not able to attach log file as it allows only image format files.

here is a sample file beat log:

2016-10-13T09:12:04Z INFO Events sent: 551
2016-10-13T09:12:04Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:11Z INFO Events sent: 534
2016-10-13T09:12:11Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:17Z INFO Events sent: 355
2016-10-13T09:12:17Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:24Z INFO Events sent: 603
2016-10-13T09:12:24Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:32Z INFO Events sent: 559
2016-10-13T09:12:32Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:39Z INFO Events sent: 542
2016-10-13T09:12:39Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:48Z INFO Events sent: 487
2016-10-13T09:12:48Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:50Z INFO Events sent: 304
2016-10-13T09:12:50Z INFO Registry file updated. 65 states written.
2016-10-13T09:12:58Z INFO Events sent: 523
2016-10-13T09:12:58Z INFO Registry file updated. 65 states written.
2016-10-13T09:13:08Z INFO Events sent: 466
2016-10-13T09:13:08Z INFO Registry file updated. 65 states written.

Sample filebeat registry log:

/log/production_2016-10-05.log":{"source":"/log/production_2016-10-05.log","offset":1446352119,"FileStateOS":{"inode":3145784,"device":51728}},"/log/production_2016-10-06.log":{"source":"/log/production_2016-10-06.log","offset":1500169936,"FileStateOS":{"inode":3145785,"device":51728}},"/log/production_2016-10-07.log":{"source":"/log/production_2016-10-07.log","offset":1466480344,"FileStateOS":{"inode":3145781,"device":51728}},"/log/production_2016-10-08.log":{"source":"/log/production_2016-10-08.log","offset":1137720684,"FileStateOS":{"inode":3145788,"device":51728}},"/log/production_2016-10-09.log":{"source":"/log/production_2016-10-09.log","offset":1097745077,"FileStateOS":{"inode":3145787,"device":51728}},"/log/production_2016-10-10.log":{"source":"/log/production_2016-10-10.log","offset":1409331639,"FileStateOS":{"inode":3145786,"device":51728}},"/log/production_2016-10-11.log":{"source":"/log/production_2016-10-11.log","offset":1333179284,"FileStateOS":{"inode":3145790,"device":51728}},"/log/production_2016-10-12.log":{"source":"/log/production_2016-10-12.log","offset":12035204,"FileStateOS":{"inode":3145789,"device":51728}},"/log/production_2016-10-13.log":{"source":"/log/production_2016-10-13.log","offset":22612575,"FileStateOS":{"inode":3145803,"device":51728}},"/var/log/apache2/access.log":{"source":"/var/log/apache2/access.log","offset":146782734,"FileStateOS":{"inode":3407877,"device":51728}}}

Please let me know if you need any other details.

12

Looking at the excerpt of the log above all looks fine. In your initial post, you state that not all events showed up. What was your method to verify that or is it currently only an assumption and that is why you want a method to verify?

13

thanks for the reply. We verify the number records against AWS ELB monitoring. In Monitoring tool we could see number of https success response which is not tallied with the indexed data shown in kibana.
For example, AWS shows there are 2500 http (200) response for a particular time. At the same time if we verify with kibana it shows nearly 2000 http (200) response.

14

@sarathymv I don't know the details of the AWS ELB monitoring, but not 100% sure if this comparison is fully reliable. It could also be that the multiline pattern is not correct. Did you find some specific events which are missing? Is the access_log file rotated?

15

I am not able to find the missing events and do not know the way to find it. Yes access_log is rotating regularly.

16

I would recommend to also have the rotated names of the access_log in your prospector config. It should work as you configured above as it will just keep the file open, but it depends on the algorithm you use for truncation.

17

This topic was automatically closed after 21 days. New replies are no longer allowed.