ES Error on logstash syslog input - Invalid date format

I am having an intermittent problem with indexing some logstash syslog
entries. It complains about an invalid date format. Oddly it has stopped
working and then seemingly started working again on its own. I obviously
may have inadvertently changed something but I have not been able to pin
down what it is. The remote host is an rsyslog ubuntu machine, I get the
errors on both stock application syslog entries and entries from our app.

Below is a sample of the error I am getting.

[2014-03-13 12:15:55,641][DEBUG][action.bulk ] [Power
Princess] [logstash-2014.03.13][3] failed to execute bulk item (index)
index {[logstash-2014.03.13][syslog][fwm304NgSEu8FRkTsh2EwQ],
source[{"message":"Lab Manager Error: Error in undeploying. Contact the
Administrator if the problem persists. on ba-labmanager02.efi.internal
when undeploying
~calculus3~617436.i18~sutirtha~watkins","@version":"1","@timestamp":"2014-03-13T19:15:39.000Z","type":"syslog","host":"calculus-daemons-2.efi.internal","priority":156,"timestamp":"Mar
13
12:15:39","logsource":"calculus-daemons-2","program":"vfi","pid":"15207","severity":4,"facility":19,"facility_label":"local3","severity_label":"Warning","tags":["Calculus","IDC"],"location":"IDC","automationserver":"labmanagerBA"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:418)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:616)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:469)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:371)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:400)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:153)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed
to parse date field [Mar 13 12:15:39], tried both date format
[dateOptionalTime], and timestamp number with locale []

at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:582)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:510)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:215)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:408)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Mar 13
12:15:39"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:576)
... 15 more

[2014-03-13 12:15:56,115][DEBUG][action.bulk ] [Power
Princess] [logstash-2014.03.13][2] failed to execute bulk item (index)
index {[logstash-2014.03.13][syslog][Pyb-uSDER0Cj7nuBQ7101Q],
source[{"message":"Deleting ~calculus3~617436.i18~sutirtha~watkins on
ba-labmanager02.efi.internal","@version":"1","@timestamp":"2014-03-13T19:15:39.000Z","type":"syslog","host":"calculus-daemons-2.efi.internal","priority":156,"timestamp":"Mar
13
12:15:39","logsource":"calculus-daemons-2","program":"vfi","pid":"15207","severity":4,"facility":19,"facility_label":"local3","severity_label":"Warning","tags":["Calculus","IDC"],"location":"IDC","automationserver":"labmanagerBA"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:418)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:616)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:469)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:371)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:400)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:153)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to
parse date field [Mar 13 12:15:39], tried both date format
[dateOptionalTime], and timestamp number with locale []
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:582)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:510)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:215)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:408)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Mar 13
12:15:39"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:576)
... 15 more
[2014-03-13 12:15:57,313][DEBUG][action.bulk ] [Power
Princess] [logstash-2014.03.13][2] failed to execute bulk item (index)
index {[logstash-2014.03.13][syslog][rhBT8L_dSsmmvFNTdlv4Bg],
source[{"message":"Lab Manager Error: Cannot delete this configuration
because one or more virtual machines inside it are currently deployed. on
ba-labmanager02.efi.internal when deleting
~calculus3~617436.i18~sutirtha~watkins","@version":"1","@timestamp":"2014-03-13T19:15:41.000Z","type":"syslog","host":"calculus-daemons-2.efi.internal","priority":156,"timestamp":"Mar
13
12:15:41","logsource":"calculus-daemons-2","program":"vfi","pid":"15207","severity":4,"facility":19,"facility_label":"local3","severity_label":"Warning","tags":["Calculus","IDC"],"location":"IDC","automationserver":"labmanagerBA"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:418)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:616)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:469)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:371)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:400)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:153)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to
parse date field [Mar 13 12:15:41], tried both date format
[dateOptionalTime], and timestamp number with locale []
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:582)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:510)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:215)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:408)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "Mar 13
12:15:41"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:576)
... 15 more

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f23cb220-1a7a-4533-b59f-d69221e16312%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You have 2 timestamp fields: @timestamp, and timestamp. Looks like the
timestamp field is the one that cannot be parsed. I see this value in the
first doc: "timestamp":"Mar 13 12:15:39". You either need to format this
properly from the LS side, or use the right date format on the ES side.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1a60d95c-f959-4f64-9307-c0aa4ce7e2f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Adding a mutate on these messages on the LS side to drop the timestamp
field did the trick. This is sort of puzzling though since that field is a
stock LS field and worked in a similar case.

Eg.

Mar 12 16:54:14 worked
Mar 13 12:59:39 failed

Thanks,

-Chris

On Thu, Mar 13, 2014 at 1:33 PM, Binh Ly binhly_es@yahoo.com wrote:

You have 2 timestamp fields: @timestamp, and timestamp. Looks like the
timestamp field is the one that cannot be parsed. I see this value in the
first doc: "timestamp":"Mar 13 12:15:39". You either need to format this
properly from the LS side, or use the right date format on the ES side.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/4msT7NJT-tM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1a60d95c-f959-4f64-9307-c0aa4ce7e2f3%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/1a60d95c-f959-4f64-9307-c0aa4ce7e2f3%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPWb6toSethsM2gs98DxHGu3h4M2EYbE2ZyAQ_%3DLHB4abnjXwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.