Filebeat log processing out of sync (not ordered by timestamp)

[pavank (Pavana Kumar N)]

Hi there,

We are using Filebeat and Logstash to collect container logs from our OCP clusters, where we have deployed Filebeat as a daemonset and Logstash is running on a VM. The logs are then routed to Azure Log Analytics Workspaces where they are configured to be aggregated in to tables.

We are seeing an issue for the logs collected from one of the application containers (multiple pods) is aggregated in Azure Log Analytics with the timestamp out of sync with a delay of upto 1-3 hours.

We want to understand if there is a way to troubleshoot the issue where Filebeat/Logstash is sending logs from one of the pods with a delay.

Attaching the screenshot of Azure Log Analytics, my Filebeat configmap and the Logstash conf files. Please do let me know if I need to updated any other details.

PS: I can confirm that the timezones of TimeGenerated and the Message in the screenshots are the same.

Name:         filebeat-config
Namespace:    kube-system
Labels:       k8s-app=filebeat
Annotations:  <none>

Data
====
filebeat.yml:
----
filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  tags: ["remrep-fb"]
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
  multiline.pattern: '(^[0-9]{4}-[0-9]{2}-[0-9]{2} |^[A-Z]\d{4} \d{2}:\d{2}:\d{2} |^[a-z]{4}\=[0-9]{4}-[0-9]{2}-[0-9]{2} |^%{MONTH})'
  multiline.negate: true
  multiline.match: after
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
#  providers:
#    - type: kubernetes
#      node: ${NODE_NAME}
#      hints.enabled: true
#      hints.default_config:
#        type: container
#        paths:
#          - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_host_metadata:

#cloud.id: ${ELASTIC_CLOUD_ID}
#cloud.auth: ${ELASTIC_CLOUD_AUTH}

#output.elasticsearch:
#  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
#  username: ${ELASTICSEARCH_USERNAME}
#  password: ${ELASTICSEARCH_PASSWORD}
output.logstash:
  hosts: ['<logstash-host>:5045']

BinaryData
====

Events:  <none>

input {
      beats {
          port => "5045"
      }
  }


output {
if [kubernetes][namespace] == "cibc-iso-prod" {
    microsoft-logstash-output-azure-loganalytics {
      workspace_id => "xxxx" # <your workspace id>
      workspace_key => "xxxx" # <your workspace key>
      custom_log_table_name => "<tablename>"
      }
    }

else if [kubernetes][namespace] == "cibc-iso-dr" {
    microsoft-logstash-output-azure-loganalytics {
      workspace_id => "xxxx" # <your workspace id>
      workspace_key => "xxxx" # <your workspace key>
      custom_log_table_name => "<tablename>"
      }
    }
}

Best Regards,
Pavan