# Filebeat log processing out of sync (not ordered by timestamp)

**URL:** <https://discuss.elastic.co/t/filebeat-log-processing-out-of-sync-not-ordered-by-timestamp/332577>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 4, 2023, 3:18pm UTC](https://discuss.elastic.co/t/filebeat-log-processing-out-of-sync-not-ordered-by-timestamp/332577 "2023-05-04T15:18:45Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![pavank](https://avatars.discourse-cdn.com/v4/letter/p/f0a364/32.png) [@pavank](https://discuss.elastic.co/u/pavank)\
**Post date:** [May 4, 2023, 3:18pm UTC](https://discuss.elastic.co/t/filebeat-log-processing-out-of-sync-not-ordered-by-timestamp/332577/1 "2023-05-04T15:18:45Z")

</div>

Hi there,

We are using Filebeat and Logstash to collect container logs from our OCP clusters, where we have deployed Filebeat as a daemonset and Logstash is running on a VM. The logs are then routed to Azure Log Analytics Workspaces where they are configured to be aggregated in to tables.

We are seeing an issue for the logs collected from one of the application containers (multiple pods) is aggregated in Azure Log Analytics with the timestamp out of sync with a delay of upto 1-3 hours.

We want to understand if there is a way to troubleshoot the issue where Filebeat/Logstash is sending logs from one of the pods with a delay.

Attaching the screenshot of Azure Log Analytics, my Filebeat configmap and the Logstash conf files. Please do let me know if I need to updated any other details.

PS: I can confirm that the timezones of TimeGenerated and the Message in the screenshots are the same.

```auto
Name: filebeat-config
Namespace: kube-system
Labels: k8s-app=filebeat
Annotations: <none>

Data
====
filebeat.yml:
----
filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  tags: ["remrep-fb"]
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
  multiline.pattern: '(^[0-9]{4}-[0-9]{2}-[0-9]{2} |^[A-Z]\d{4} \d{2}:\d{2}:\d{2} |^[a-z]{4}\=[0-9]{4}-[0-9]{2}-[0-9]{2} |^%{MONTH})'
  multiline.negate: true
  multiline.match: after
# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
#filebeat.autodiscover:
# providers:
# - type: kubernetes
# node: ${NODE_NAME}
# hints.enabled: true
# hints.default_config:
# type: container
# paths:
# - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_host_metadata:

#cloud.id: ${ELASTIC_CLOUD_ID}
#cloud.auth: ${ELASTIC_CLOUD_AUTH}

#output.elasticsearch:
# hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
# username: ${ELASTICSEARCH_USERNAME}
# password: ${ELASTICSEARCH_PASSWORD}
output.logstash:
  hosts: ['<logstash-host>:5045']

BinaryData
====

Events: <none>

```

```auto
input {
      beats {
          port => "5045"
      }
  }

output {
if [kubernetes][namespace] == "cibc-iso-prod" {
    microsoft-logstash-output-azure-loganalytics {
      workspace_id => "xxxx" # <your workspace id>
      workspace_key => "xxxx" # <your workspace key>
      custom_log_table_name => "<tablename>"
      }
    }

else if [kubernetes][namespace] == "cibc-iso-dr" {
    microsoft-logstash-output-azure-loganalytics {
      workspace_id => "xxxx" # <your workspace id>
      workspace_key => "xxxx" # <your workspace key>
      custom_log_table_name => "<tablename>"
      }
    }
}

```

 ![Log Sync issue between Filebeat and Log Analytics Workspaces 1](https://us1.discourse-cdn.com/elastic/original/3X/0/3/034665c3bc933826d5f0921b629ed54ef58d764b.png)  
 ![Log Sync issue between Filebeat and Log Analytics Workspaces 2](https://us1.discourse-cdn.com/elastic/original/3X/a/e/ae9121ddec9f3e215d4bf6aaf58efe6050572753.png)

Best Regards,  
Pavan

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 1, 2023, 5:19pm UTC](https://discuss.elastic.co/t/filebeat-log-processing-out-of-sync-not-ordered-by-timestamp/332577/2 "2023-06-01T17:19:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.