Having troubles parsing dates

Ben_Morrice · November 25, 2013, 10:36am

Hello,

I have a basic logstash -> elasticsearch instance running to parse apache
logs; or at least I did. I don't know what has changed but elasticsearch is
now bailing on my timestamp field and is not able to correctly parse it.

I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this case)

Is anyone able to assist me? I'm sure this is a simple fix?!

Please see below an example from my elasticsearch log

[2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] "GET
/examplerequest HTTP/1.1" 200 11415 "-" "Jakarta
Commons-HttpClient/3.1"","@timestamp":"2013-11-25T08:49:47.548Z","@version":"1","type":"webfrontend","file":"/var/loghttpdssl_access_log","host":"example.com","offset":"629386952","clientip":"10.80.5.30","ident":"-","auth":"-","timestamp":"25/Nov/2013:09:49:45
+0100","verb":"GET","request":"/examplerequest","httpversion":"1.1","response":"200","bytes":"11415","referrer":""-"","agent":""Jakarta
Commons-HttpClient/3.1"","tags":["apache"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:398)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:618)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:471)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:513)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:457)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:342)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to
parse date field [25/Nov/2013:09:49:45 +0100], tried both date format
[dateOptionalTime], and timestamp number with locale []
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:486)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:424)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:188)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:387)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format:
"25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:480)
... 15 more

Thanks in advance!

Cheers,

Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Phil_Dougherty · December 11, 2013, 8:26pm

I'm having the same exact issue. Running on a 4 node cluster and using ES
0.90.3. Strangely this issue will randomly bite me, and it seems like it
happens when the index is living on a certain server. Were you able to
resolve this issue or could you provide some more information that may help
me resolve this?

On Monday, November 25, 2013 5:36:41 AM UTC-5, Ben Morrice wrote:

Hello,

I have a basic logstash -> elasticsearch instance running to parse apache
logs; or at least I did. I don't know what has changed but elasticsearch is
now bailing on my timestamp field and is not able to correctly parse it.

I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this case)

Is anyone able to assist me? I'm sure this is a simple fix?!

Please see below an example from my elasticsearch log

[2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] "GET
/examplerequest HTTP/1.1" 200 11415 "-" "Jakarta
Commons-HttpClient/3.1"","@timestamp":"2013-11-25T08:49:47.548Z","@version":"1","type":"webfrontend","file":"/var/loghttpdssl_access_log","host":"
example.com","offset":"629386952","clientip":"10.80.5.30","ident":"-","auth":"-","timestamp":"25/Nov/2013:09:49:45
+0100","verb":"GET","request":"/examplerequest","httpversion":"1.1","response":"200","bytes":"11415","referrer":""-"","agent":""Jakarta
Commons-HttpClient/3.1"","tags":["apache"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:398)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:618)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:471)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:513)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:457)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:342)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed
to parse date field [25/Nov/2013:09:49:45 +0100], tried both date format
[dateOptionalTime], and timestamp number with locale
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:486)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:424)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:188)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:387)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format:
"25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:480)
... 15 more

Thanks in advance!

Cheers,

Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/519c7a1a-ac51-46a8-8d02-88291da9b1be%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ben_Morrice · December 12, 2013, 9:29am

Hi Phil,

Unfortunately i'm still seeing the same issue. Like your case this issue
comes and goes as well. I am running a 2 node cluster (ES 0.90.7) with
default configuration. I'm using logstash-1.2.2; however looking at the
debug logs within logstash the events are parsed correctly, they just die
when elasticsearch takes the floor.

Sorry I can't be of any more help... If you have any theories, please share
your thoughts.

Cheers,

Ben

On Wednesday, 11 December 2013 21:26:41 UTC+1, Phil Dougherty wrote:

I'm having the same exact issue. Running on a 4 node cluster and using ES
0.90.3. Strangely this issue will randomly bite me, and it seems like it
happens when the index is living on a certain server. Were you able to
resolve this issue or could you provide some more information that may help
me resolve this?

On Monday, November 25, 2013 5:36:41 AM UTC-5, Ben Morrice wrote:

Hello,

I have a basic logstash -> elasticsearch instance running to parse apache
logs; or at least I did. I don't know what has changed but elasticsearch is
now bailing on my timestamp field and is not able to correctly parse it.

I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this
case)

Is anyone able to assist me? I'm sure this is a simple fix?!

Please see below an example from my elasticsearch log

[2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] "GET
/examplerequest HTTP/1.1" 200 11415 "-" "Jakarta
Commons-HttpClient/3.1"","@timestamp":"2013-11-25T08:49:47.548Z","@version":"1","type":"webfrontend","file":"/var/loghttpdssl_access_log","host":"
example.com","offset":"629386952","clientip":"10.80.5.30","ident":"-","auth":"-","timestamp":"25/Nov/2013:09:49:45
+0100","verb":"GET","request":"/examplerequest","httpversion":"1.1","response":"200","bytes":"11415","referrer":""-"","agent":""Jakarta
Commons-HttpClient/3.1"","tags":["apache"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:398)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:618)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:471)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:513)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:457)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:342)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed
to parse date field [25/Nov/2013:09:49:45 +0100], tried both date format
[dateOptionalTime], and timestamp number with locale
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:486)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:424)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:188)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:387)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format:
"25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:480)
... 15 more

Thanks in advance!

Cheers,

Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/087f7e84-261c-4bde-b214-9be3580edae1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

spinscale · December 12, 2013, 10:01am

Hey,

judging from a quick peek at the problem: The value you are trying to parse
as the timestamp field (note: not the logstash @timestamp field) is an
unknown date format. This means it will only happen with apache access
logs. Somehow the timestamp field has already been configured to be of a
different format (from a past log entry most likely).

Two possibilites (note: maybe there are others, I am not a logstash wizard)

Make the timestamp field a string and do not analyze it (which also
means, you cannot search for it, but usually you are using the @timestamp
field for that anyway)
Have an own type for the apache access logs in your logstash config,
which uses a special timestamp based mapping format, that fits to CLF
timestamps

See

--Alex

On Thu, Dec 12, 2013 at 10:29 AM, Ben Morrice ben.morrice@gmail.com wrote:

Hi Phil,

Unfortunately i'm still seeing the same issue. Like your case this issue
comes and goes as well. I am running a 2 node cluster (ES 0.90.7) with
default configuration. I'm using logstash-1.2.2; however looking at the
debug logs within logstash the events are parsed correctly, they just die
when elasticsearch takes the floor.

Sorry I can't be of any more help... If you have any theories, please
share your thoughts.

Cheers,

Ben

On Wednesday, 11 December 2013 21:26:41 UTC+1, Phil Dougherty wrote:

I'm having the same exact issue. Running on a 4 node cluster and using ES
0.90.3. Strangely this issue will randomly bite me, and it seems like it
happens when the index is living on a certain server. Were you able to
resolve this issue or could you provide some more information that may help
me resolve this?

On Monday, November 25, 2013 5:36:41 AM UTC-5, Ben Morrice wrote:

Hello,

I have a basic logstash -> elasticsearch instance running to parse
apache logs; or at least I did. I don't know what has changed but
elasticsearch is now bailing on my timestamp field and is not able to
correctly parse it.

I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this
case)

Is anyone able to assist me? I'm sure this is a simple fix?!

Please see below an example from my elasticsearch log

[2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] "GET
/examplerequest HTTP/1.1" 200 11415 "-" "Jakarta
Commons-HttpClient/3.1"","@timestamp":"2013-11-25T08:49:
47.548Z","@version":"1","type":"webfrontend","file":"/var/
loghttpdssl_access_log","host":"example.com","offset":"
629386952","clientip":"10.80.5.30","ident":"-","auth":"-","
timestamp":"25/Nov/2013:09:49:45 +0100","verb":"GET","request":
"/examplerequest","httpversion":"1.1","response":"200","bytes":"11415","
referrer":""-"","agent":""Jakarta Commons-HttpClient/3.1"","
tags":["apache"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:398)
at org.elasticsearch.index.mapper.object.ObjectMapper.
serializeValue(ObjectMapper.java:618)
at org.elasticsearch.index.mapper.object.ObjectMapper.
parse(ObjectMapper.java:471)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:513)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:457)
at org.elasticsearch.index.shard.service.InternalIndexShard.
prepareCreate(InternalIndexShard.java:342)
at org.elasticsearch.action.bulk.TransportShardBulkAction.
shardIndexOperation(TransportShardBulkAction.java:401)
at org.elasticsearch.action.bulk.TransportShardBulkAction.
shardOperationOnPrimary(TransportShardBulkAction.java:155)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction.
performOnPrimary(TransportShardReplicationOperationAction.java:556)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction$1.
run(TransportShardReplicationOperationAction.java:426)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException:
failed to parse date field [25/Nov/2013:09:49:45 +0100], tried both date
format [dateOptionalTime], and timestamp number with locale
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(
DateFieldMapper.java:486)
at org.elasticsearch.index.mapper.core.DateFieldMapper.
innerParseCreateField(DateFieldMapper.java:424)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.
parseCreateField(NumberFieldMapper.java:188)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:387)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format:
"25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.
parseMillis(DateTimeFormatter.java:754)
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(
DateFieldMapper.java:480)
... 15 more

Thanks in advance!

Cheers,

Ben

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/087f7e84-261c-4bde-b214-9be3580edae1%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_eOC8rpT82U_4AQ3GNX93BWvRPZkP9oe1j1q0unRu_BA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.