# How to find lag between filebeat and logstash based on timestamp

**URL:** <https://discuss.elastic.co/t/how-to-find-lag-between-filebeat-and-logstash-based-on-timestamp/100391>\
**Category:** Logstash\
**Created:** [September 13, 2017, 4:58pm UTC](https://discuss.elastic.co/t/how-to-find-lag-between-filebeat-and-logstash-based-on-timestamp/100391 "2017-09-13T16:58:11Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gangadhar\_Mahadevan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gangadhar_mahadevan/32/62579_2.png) [@Gangadhar\_Mahadevan](https://discuss.elastic.co/u/Gangadhar_Mahadevan)\
**Post date:** [September 13, 2017, 4:58pm UTC](https://discuss.elastic.co/t/how-to-find-lag-between-filebeat-and-logstash-based-on-timestamp/100391/1 "2017-09-13T16:58:12Z")

</div>

Hi,

I am trying to find lag between filebeat sending a message and time at which logstash reads and process it from topic and not sure how to achieve it.

This is snippet of my filebeat config

## filebeat\_prospectors:

```
paths:
  - '/var/log/syslog'
  - '/var/log/auth.log'
  - '/var/log/messages'
input_type: log
fields:
  source: filebeat
  format: syslog
document_type: "{{ stack_env }}-{{ datacenter }}-{{ datacenter_type }}-syslog"

```

And we have logstash runners in kafka topic with input, filter and output configration

below is sample message I read from syslog topic,

{"@timestamp":"2017-09-11T21:17:02.778Z","beat":{"hostname":"xxx","name":"xxx","version":"5.2.0"},"fields":{"format":"syslog","source":"filebeat"},"input\_type":"log","message":"Sep 11 21:17:01 xxx)","offset":,"source":"/var/log/syslog","type":"xx-syslog"}

I had a filter like  
kv {  
field\_split =\> ","  
}  
ruby {  
code =\> "

# event.to\_hash.each\_pair{|k,v|

# if k.include? '@timestamp'

```
         event.set('access_input_filebeat_time', event.get('@timestamp'))
        }
    "
   }
}

```

to capture @timestamp and my understanding was the @timestamp field when I read message is time at which filebeat sent the log

I am also aware logstash creates @timestamp field when it parses log. But when I look at kibana the @timestamp(which I believe logstash creates) and new filed access\_input\_filebeat\_time has same value.

My goal is to find the lag between time at which filebeat read the message from /var/log file and time at which logstahs runner received and processed it and capture them in 2 different field in kibana. Kindly advise. Thanks?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2017, 4:58pm UTC](https://discuss.elastic.co/t/how-to-find-lag-between-filebeat-and-logstash-based-on-timestamp/100391/2 "2017-10-11T16:58:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.