Filebeat lagging behind raw logs

Hey Everyone,

Recently primary ingest node for network devices syslogs, (unifi APs, cisco switches, sonicwalls, etc) has been lagging behind the raw logs. In my setup I'm using rsyslog to write the initial logs to file first, then having filebeat scrape those and ship them to another server running elasticsearch. I thought it may be an issue with the collector node, so I shut that down, redirected all the syslogs to the primary elasticsearch node, then collected them using rsyslog and shipped using filebeat, the same as the collector, and still had the same issue.

The strange part is that it only affects my high log count sources, and to varying degrees. For example, my sonicwall logs will catch up to within a few minutes of live during the hours that there's not much traffic. During the day when there's a lot more, they'll lag up to a few hours behind. Unifi syslogs are much fewer and only take a few seconds between being sent and being ingested. Same goes for dhcp logs shipped by filebeat running on another windows server, cisco ios and sbs logs using a custom pipeline and shipped from the ingest/primary server, and cisco asa logs using the built in module in filebeat, also shipped from the ingest/primary node.

Cisco FTD logs, using the filebeat module are the worst of everything. at 10am, I have from midnight to 4AM. It takes 2-3 days to fully ingest 1 days FTD logs

Here's my configs:

Filebeat:

    filebeat.inputs:
     - type: log
       paths:
        - /logs/syslog/sbs/*/*.log
     - type: log
       paths:
        - /logs/syslog/ios/*/*.log
     - type: log
       paths:
        - /logs/syslog/sonicwall/*/*.log
     - type: log
       paths:
        - /logs/syslog/unifi/*/*.log
     - type: log
       paths:
        - /logs/filebeat/*

    filebeat.config.modules:
     path: /etc/filebeat/modules.d/*.yml
     reload.enabled: true

    logging.level: info
    logging.to_files: true
    logging.files:
      path: /logs/filebeat/
      name: filebeat
      keepfiles: 7
      permissions: 0644

    monitoring.enabled: true


    http.enabled: true
    http.host: localhost
    http.port: 5066

    output.elasticsearch.ssl.certificate_authorities: ["/etc/filebeat/elasticsearch-ca.pem"]
    output.elasticsearch:
      hosts: ['https://<ES-IP>:9200']
      username: "--"
      password: "--"
      pipelines:
        - pipeline: "sbs-pipeline"
          when.contains:
            log.file.path: "/logs/syslog/sbs"
        - pipeline: ios-pipeline
          when.contains:
            log.file.path: "/logs/syslog/ios"
        - pipeline: sonicwall-pipeline
          when.contains:
            log.file.path: "/logs/syslog/sonicwall"
        - pipeline: ubiquiti-ap-log-pipeline
          when.contains:
            log.file.path: "/logs/syslog/unifi"
        - pipeline: remote-filebeat-pipeline
          when.contains:
            log.file.path: "/logs/filebeat"

    setup.kibana:
      host: "https://<ES-IP>:5601"
      ssl:
        enabled: true
        verification_mode: none

    setup.ilm.enabled: false
    setup.template.enabled: false
    setup.template.name: "filebeat"
    setup.template.pattern: "filebeat-*"

Elasticsearch:

    # ======================== Elasticsearch Configuration =========================
    #
    # NOTE: Elasticsearch comes with reasonable defaults for most settings. Before you set out to tweak and tune the
    #       configuration, make sure you understand what are you trying to accomplish and the consequences.
    #
    # The primary way of configuring a node is via this file. This template lists the most important settings you may
    # want to configure for a production cluster.
    #
    # Please consult the documentation for further information on configuration options:
    # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
    #
    script.context.processor_conditional.max_compilations_rate: 50000/1m
    script.painless.regex.enabled: true
    script.context.template.max_compilations_rate: 50000/1m
    # ---------------------------------- Cluster -----------------------------------
    #
    # Use a descriptive name for your cluster:
    #
    cluster.name: --
    #
    # ------------------------------------ Node ------------------------------------
    #
    # Use a descriptive name for the node:
    #
    node.name: --
    #
    # Add custom attributes to the node:
    #
    #node.attr.rack: r1
    #
    # ----------------------------------- Paths ------------------------------------
    #
    # Path to directory where to store the data (separate multiple locations by comma):
    #
    path.data: /logs/elasticsearch/data
    #
    # Path to log files:
    #
    path.logs: /logs/elasticsearch/
    #
    # ----------------------------------- Memory -----------------------------------
    #
    # Lock the memory on startup:
    #
    #bootstrap.memory_lock: true
    #
    # Make sure that the heap size is set to about half the memory available on the system and that the owner of the
    # process is allowed to use this limit.
    #
    # Elasticsearch performs poorly when the system is swapping the memory.
    #
    # ---------------------------------- Network -----------------------------------
    #
    # Set the bind address to a specific IP (IPv4 or IPv6):
    #
    network.host: <ES-IP>
    #
    # Set a custom port for HTTP:
    #
    #http.port: 9200
    #
    # For more information, consult the network module documentation.
    #
    # --------------------------------- Discovery ----------------------------------
    #
    # Pass an initial list of hosts to perform discovery when this node is started: The default list of hosts is
    # ["127.0.0.1", "[::1]"]
    #
    #discovery.seed_hosts: ["host1", "host2"]
    #
    # Bootstrap the cluster using an initial set of master-eligible nodes:
    #
    cluster.initial_master_nodes: ["--"]
    #
    # For more information, consult the discovery and cluster formation module documentation.
    #
    # ---------------------------------- Gateway -----------------------------------
    #
    # Block initial recovery after a full cluster restart until N nodes are started:
    #
    #gateway.recover_after_nodes: 3
    #
    # For more information, consult the gateway module documentation.
    #
    # ---------------------------------- Various -----------------------------------
    #
    # Require explicit names when deleting indices:
    #
    #action.destructive_requires_name: true
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.keystore.path: /--
    xpack.security.audit.enabled: true
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: "--"
    xpack.security.authc.api_key.enabled: true

    xpack.notification.email.account:
      exchange_account:
        profile: outlook
        email_defaults:
          from: --
        smtp:
          host: --
          port: 25

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.