i've been trying to add the "@received_at" timestamp that indicates when logstash received the log message. this helps us determine if our logging infrastructure is lagging behind our logging events. The logs we receive already have a "@timestamp" field so using the typical assignment of "@received_at" to "@timestamp" will not work. I'm not seeing anything in the documentation about having the ability to do this. So question is it possible to add the current time to a an arbitrary field w/o referencing an existing field?
If I understand the ask correctly, then this should do it...
ruby { code => "event.set('[rightNow]', Time.now)" }
But logstash is adding @timestamp, right? If there is latency upstream I do not think this is going to help you find it.
1 Like
Assigning the received timestamp in Logstash may not always be entirely accurate, as it is possible for Logstash to process an event and then have to retry sending it to Elasticsearch a number of times. It is however likely that only a limited number of events would be be affected by this. Another option might be to assign it in an ingest pipeline once it reaches Elasticsearch, as shown in this example.
logstash is not setting @timestamp, it is already set before being received by logstash. I will look into your ruby option. Thank you for your help.
i'm not looking for the time that ES received the data but rather when logstash received the data. we often have occasional delays in log processing and having this bit of information helps us know where the problem lies.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.