How to find lag between filebeat and logstash based on timestamp

[Gangadhar_Mahadevan]

Hi,

I am trying to find lag between filebeat sending a message and time at which logstash reads and process it from topic and not sure how to achieve it.

This is snippet of my filebeat config

filebeat_prospectors:

paths:
  - '/var/log/syslog'
  - '/var/log/auth.log'
  - '/var/log/messages'
input_type: log
fields:
  source: filebeat
  format: syslog
document_type: "{{ stack_env }}-{{ datacenter }}-{{ datacenter_type }}-syslog"

And we have logstash runners in kafka topic with input, filter and output configration

below is sample message I read from syslog topic,

{"@timestamp":"2017-09-11T21:17:02.778Z","beat":{"hostname":"xxx","name":"xxx","version":"5.2.0"},"fields":{"format":"syslog","source":"filebeat"},"input_type":"log","message":"Sep 11 21:17:01 xxx)","offset":,"source":"/var/log/syslog","type":"xx-syslog"}

I had a filter like
kv {
field_split => ","
}
ruby {
code => "

event.to_hash.each_pair{|k,v|

if k.include? '@timestamp'

         event.set('access_input_filebeat_time', event.get('@timestamp'))
        }
    "
   }
}

to capture @timestamp and my understanding was the @timestamp field when I read message is time at which filebeat sent the log

I am also aware logstash creates @timestamp field when it parses log. But when I look at kibana the @timestamp(which I believe logstash creates) and new filed access_input_filebeat_time has same value.

My goal is to find the lag between time at which filebeat read the message from /var/log file and time at which logstahs runner received and processed it and capture them in 2 different field in kibana. Kindly advise. Thanks?