Date match failure failure with given format

Elastic Stack Logstash
1

Hi,

I am getting below error in ES logs.
This is related to the match failure. I don't how it appears in ES?

[2017-05-08 10:30:19,469][WARN ][indices.cluster          ] [data-node-3] [[esb-2017.05.08][0]] marking and sending shard failed due to [engine failure, reason [indices:data/write/bulk[s] failed on replica]]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [eventLogTime]
  at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:409)
    at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:706)
    at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:497)
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:544)
    at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
    at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:453)
    at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:580)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:249)
    at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$ReplicaOperationTransportHandler.messageReceived(TransportShardReplicationOperationAction.java:228)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [5/8/17 10:30:11:094 EEST], tried both date format [dateOptionalTime], and timestamp number with locale []
    at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:619)
    at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:547)
    at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:236)
    at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:399)
    ... 13 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "5/8/17 10:30:11:094 EEST" is malformed at "/8/17 10:30:11:094 EEST"
    at org.elasticsearch.common.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)
    at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)
    at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:613)
    ... 16 more

Log data is :[5/8/17 10:38:55:422 EEST] 00000035 JDBCRAResourc I JDBCManagedConnection getWBIConnection Database connection in 'DataSource JNDI' mode is created successfully!

I have date match filter in logstash:
match => [ "eventLogTime", "YYYY-MM-dd'T'HH:mm:ss", "YYYY-MM-dd HH:mm:ss","HH:mm:ss MMM dd yyyy","YYYY-MM-dd HH:mm:ss,SSS","yyyy.MM.dd G 'at' HH:mm:ss z","yyyyy.MMMMM.dd GGG hh:mm aaa","EEE, d MMM yyyy HH:mm:ss Z","yyyy-MM-dd'T'HH:mm:ss.SSSZ","yyyy-MM-dd'T'HH:mm:ss.SSSZ+0300","YYYY-MM-dd HH:mm:ss.SSS","YYYY-MM-dd HH:mm:ss.S","dd/MMM/yyyy:HH:mm:ss Z","YYYY-MM-dd'T'HH:mm:ss+02:00","yyyy/MM/dd HH:mm:ss","YYYY-MM-dd'T'HH:mm:ss+03:00","ISO8601","YYYY-MM-dd-HH:mm:ss.SSSZ","MMM dd HH:mm:ss","dd.MM.YYYY HH:mm:ss:SSS"]

I don't know what pattern to add in the date match filter.

2

The date filter can't parse timezone names like EEST, but you can use a translate filter to map such strings to UTC offsets. Or, if the log event's timezone always matches the timezone of the machine where Logstash runs you can just skip the timezone specifier.

Apart from that it should be straight forward. These date patterns will be able to parse the date when the timezone is gone:

  • M/d/YY HH:mm:ss:SSS
  • M/dd/YY HH:mm:ss:SSS
  • MM/d/YY HH:mm:ss:SSS
  • MM/dd/YY HH:mm:ss:SSS
3

Hi

Log event timezone and logstash server are in same timzone.

Does it mean I need to add above timestamp pattern in date filter and I don't need to use translate filter?
Or I have to use translate filter to skip timzone specifier?

br,
Sunil.

4

Does it mean I need to add above timestamp pattern in date filter and I don't need to use translate filter?

Those patterns assume that the timestamp being parsed doesn't include a timezone.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.