Filebeat add system time in prospectors field

[Gangadhar_Mahadevan]

Filebeat version 5.2

Is there a way to add system current time (using date command) as a custom field in filebeat prosectors? I am able to add hardcoded values in custom fields but am looking for gathering system time as custom fields to capture at what time filebeat ships the log messages.

Something like

filebeat_prospectors:

path : /var/log/syslog
fields:
source: filebeat
time: {current time}

Thanks!!

[steffens (Steffen Siering)]

The @timestamp field is exactly the time filebeat did read the log line. Lines might be buffered for unknown time, due to backpressure. E.g. in Logstash you can get the arrival time and this way compute how long it took for the event being ready for shipping until the event is finally processed by Logstash.

[Gangadhar_Mahadevan]

This is sample message that has @timestamp when I read from kafka topic. I tried to extract @timestamp in different logstash filters but was unsuccessful.
Sample message:
{"@timestamp":"2017-09-11T21:17:04.020Z","beat":{"hostname":"ossmlpadc0101a","name":"ossmlpadc0101a","version":"5.2.0"},"fields":{"@timestamp":"2017-09-11T21:17:02.778Z","beat":{"hostname":"xxxx,"name":"xxxx","version":"5.2.0"},"fields":{"format":"syslog","source":"filebeat"},"input_type":"log","source":"/var/log/syslog","type":"local-dev-vergil-adc01-rg-
syslog

Tried ruby plugin answer from Get current time using ruby filter in logstash

Also Tried date filter plugin and didn't help.

The error message is
Caused by: java.lang.IllegalStateException: Pattern does not contain a date
at org.apache.logging.log4j.core.appender.rolling.PatternProcessor.getNextTime(PatternProcessor.java:119)
at org.apache.logging.log4j.core.appender.rolling.TimeBasedTriggeringPolicy.initialize(TimeBasedTriggeringPolicy.java:59)
at org.apache.logging.log4j.core.appender.rolling.CompositeTriggeringPolicy.initialize(CompositeTriggeringPolicy.java:49)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.initialize(RollingFileManager.java:82)
at org.apache.logging.log4j.core.appender.RollingFileAppender.createAppender(RollingFileAppender.java:197)
... 108 more

2017-09-11 17:14:19,755 main ERROR Null object returned for RollingFile in Appenders.

Am trying to find how long it took filebeat to ship message and logstash to process and forward to Elasticsearch or another kafka topic. Thanks!!

[steffens (Steffen Siering)]

I moved this topic to the Logstash forum. Problem is about configuring logstash to parse and process timestamps.

[Gangadhar_Mahadevan]

Thanks!! Anyone from logstash team can help with this? This seems pretty simple but for some reasons whatever methods I try it says illegal date pattern