Hey Everyone,
Recently primary ingest node for network devices syslogs, (unifi APs, cisco switches, sonicwalls, etc) has been lagging behind the raw logs. In my setup I'm using rsyslog to write the initial logs to file first, then having filebeat scrape those and ship them to another server running elasticsearch. I thought it may be an issue with the collector node, so I shut that down, redirected all the syslogs to the primary elasticsearch node, then collected them using rsyslog and shipped using filebeat, the same as the collector, and still had the same issue.
The strange part is that it only affects my high log count sources, and to varying degrees. For example, my sonicwall logs will catch up to within a few minutes of live during the hours that there's not much traffic. During the day when there's a lot more, they'll lag up to a few hours behind. Unifi syslogs are much fewer and only take a few seconds between being sent and being ingested. Same goes for dhcp logs shipped by filebeat running on another windows server, cisco ios and sbs logs using a custom pipeline and shipped from the ingest/primary server, and cisco asa logs using the built in module in filebeat, also shipped from the ingest/primary node.
Cisco FTD logs, using the filebeat module are the worst of everything. at 10am, I have from midnight to 4AM. It takes 2-3 days to fully ingest 1 days FTD logs
Here's my configs:
Filebeat:
filebeat.inputs:
- type: log
paths:
- /logs/syslog/sbs/*/*.log
- type: log
paths:
- /logs/syslog/ios/*/*.log
- type: log
paths:
- /logs/syslog/sonicwall/*/*.log
- type: log
paths:
- /logs/syslog/unifi/*/*.log
- type: log
paths:
- /logs/filebeat/*
filebeat.config.modules:
path: /etc/filebeat/modules.d/*.yml
reload.enabled: true
logging.level: info
logging.to_files: true
logging.files:
path: /logs/filebeat/
name: filebeat
keepfiles: 7
permissions: 0644
monitoring.enabled: true
http.enabled: true
http.host: localhost
http.port: 5066
output.elasticsearch.ssl.certificate_authorities: ["/etc/filebeat/elasticsearch-ca.pem"]
output.elasticsearch:
hosts: ['https://<ES-IP>:9200']
username: "--"
password: "--"
pipelines:
- pipeline: "sbs-pipeline"
when.contains:
log.file.path: "/logs/syslog/sbs"
- pipeline: ios-pipeline
when.contains:
log.file.path: "/logs/syslog/ios"
- pipeline: sonicwall-pipeline
when.contains:
log.file.path: "/logs/syslog/sonicwall"
- pipeline: ubiquiti-ap-log-pipeline
when.contains:
log.file.path: "/logs/syslog/unifi"
- pipeline: remote-filebeat-pipeline
when.contains:
log.file.path: "/logs/filebeat"
setup.kibana:
host: "https://<ES-IP>:5601"
ssl:
enabled: true
verification_mode: none
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
Elasticsearch:
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings. Before you set out to tweak and tune the
# configuration, make sure you understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists the most important settings you may
# want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
script.context.processor_conditional.max_compilations_rate: 50000/1m
script.painless.regex.enabled: true
script.context.template.max_compilations_rate: 50000/1m
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: --
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: --
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /logs/elasticsearch/data
#
# Path to log files:
#
path.logs: /logs/elasticsearch/
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available on the system and that the owner of the
# process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: <ES-IP>
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started: The default list of hosts is
# ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["--"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: /--
xpack.security.audit.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "--"
xpack.security.authc.api_key.enabled: true
xpack.notification.email.account:
exchange_account:
profile: outlook
email_defaults:
from: --
smtp:
host: --
port: 25