This is sample message that has @timestamp when I read from kafka topic. I tried to extract @timestamp in different logstash filters but was unsuccessful.
Sample message:
{"@timestamp":"2017-09-11T21:17:04.020Z","beat":{"hostname":"ossmlpadc0101a","name":"ossmlpadc0101a","version":"5.2.0"},"fields":{"@timestamp":"2017-09-11T21:17:02.778Z","beat":{"hostname":"xxxx,"name":"xxxx","version":"5.2.0"},"fields":{"format":"syslog","source":"filebeat"},"input_type":"log","source":"/var/log/syslog","type":"local-dev-vergil-adc01-rg-
syslog
Tried ruby plugin answer from Get current time using ruby filter in logstash
Also Tried date filter plugin and didn't help.
The error message is
Caused by: java.lang.IllegalStateException: Pattern does not contain a date
at org.apache.logging.log4j.core.appender.rolling.PatternProcessor.getNextTime(PatternProcessor.java:119)
at org.apache.logging.log4j.core.appender.rolling.TimeBasedTriggeringPolicy.initialize(TimeBasedTriggeringPolicy.java:59)
at org.apache.logging.log4j.core.appender.rolling.CompositeTriggeringPolicy.initialize(CompositeTriggeringPolicy.java:49)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.initialize(RollingFileManager.java:82)
at org.apache.logging.log4j.core.appender.RollingFileAppender.createAppender(RollingFileAppender.java:197)
... 108 more
2017-09-11 17:14:19,755 main ERROR Null object returned for RollingFile in Appenders.
Am trying to find how long it took filebeat to ship message and logstash to process and forward to Elasticsearch or another kafka topic. Thanks!!