Delay in metrics leaving logstash

Elastic Stack Beats
1

Hi,
I'm running Filebeats 5.6.1 and logstash 2.2 on Windows server 2012.

(This was all setup by somebody who has left the company)

We use filebeats to collect the IIS http request log files on a 4 webservers. which go to a single logstash process, then to graphite and some of the metrics go off to elasticsearch.

When we restart the filebeat service on one of the servers, it takes around 4 hours for the metrics to start coming out of the logstash pipeline again. The rest of the metrics carry on coming out of logstash from the other servers as expected.

The logfiles for both Logstash and Filebeat have nothing interesting in them.

could I get some guidence on how to better uncover the delay?

There are two prospectors in the filebeat config, here is the top one:

filebeat:
  prospectors:
    -
      paths:
        - d:\IISLogs\Logfiles\W3SVC1294154473\*.log
      input_type: log
      fields:
        log_source: IIS
        platform: production
        app: sharepoint
        webapp: worksites
      fields_under_root: true
      ignore_older: 2h
      tail_files: true

    -
      paths:
        - d:\IISLogs\Logfiles\W3SVC562970453\*.log
      input_type: log
      fields:
        log_source: IIS
        platform: production
        app: sharepoint
        webapp: quantum
      fields_under_root: true
      ignore_older: 2h
      tail_files: true

  registry_file: "C:/ProgramData/filebeat/registry"

output:
  logstash: 
    hosts: ["xxx.xxx.com:5088"] # hidden

############################# Shipper #########################################
shipper:
############################# Logging #########################################
logging:
  files:
    path: logs
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 4
2

The results seem to be sent through in big batches, with big delays between sending. Any pointers for me to debug this?

For example I restarted the filebeat at around 16:00. Nothing happened until 23:00 where 250K messages suddenly came through, then it was quiet, then at 01:06 465K messages came through. Now all is quiet again.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.