I have a basic logstash -> elasticsearch instance running to parse apache
logs; or at least I did. I don't know what has changed but elasticsearch is
now bailing on my timestamp field and is not able to correctly parse it.
I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this case)
Is anyone able to assist me? I'm sure this is a simple fix?!
Please see below an example from my elasticsearch log
[2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] "GET
/examplerequest HTTP/1.1" 200 11415 "-" "Jakarta
Commons-HttpClient/3.1"","@timestamp":"2013-11-25T08:49:47.548Z","@version":"1","type":"webfrontend","file":"/var/loghttpdssl_access_log","host":"example.com","offset":"629386952","clientip":"10.80.5.30","ident":"-","auth":"-","timestamp":"25/Nov/2013:09:49:45
+0100","verb":"GET","request":"/examplerequest","httpversion":"1.1","response":"200","bytes":"11415","referrer":""-"","agent":""Jakarta
Commons-HttpClient/3.1"","tags":["apache"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:398)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:618)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:471)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:513)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:457)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:342)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to
parse date field [25/Nov/2013:09:49:45 +0100], tried both date format
[dateOptionalTime], and timestamp number with locale []
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:486)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:424)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:188)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:387)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format:
"25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:480)
... 15 more
I'm having the same exact issue. Running on a 4 node cluster and using ES
0.90.3. Strangely this issue will randomly bite me, and it seems like it
happens when the index is living on a certain server. Were you able to
resolve this issue or could you provide some more information that may help
me resolve this?
On Monday, November 25, 2013 5:36:41 AM UTC-5, Ben Morrice wrote:
Hello,
I have a basic logstash -> elasticsearch instance running to parse apache
logs; or at least I did. I don't know what has changed but elasticsearch is
now bailing on my timestamp field and is not able to correctly parse it.
I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this case)
Is anyone able to assist me? I'm sure this is a simple fix?!
Please see below an example from my elasticsearch log
[2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] "GET
/examplerequest HTTP/1.1" 200 11415 "-" "Jakarta
Commons-HttpClient/3.1"","@timestamp":"2013-11-25T08:49:47.548Z","@version":"1","type":"webfrontend","file":"/var/loghttpdssl_access_log","host":" example.com","offset":"629386952","clientip":"10.80.5.30","ident":"-","auth":"-","timestamp":"25/Nov/2013:09:49:45
+0100","verb":"GET","request":"/examplerequest","httpversion":"1.1","response":"200","bytes":"11415","referrer":""-"","agent":""Jakarta
Commons-HttpClient/3.1"","tags":["apache"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:398)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:618)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:471)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:513)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:457)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:342)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed
to parse date field [25/Nov/2013:09:49:45 +0100], tried both date format
[dateOptionalTime], and timestamp number with locale
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:486)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:424)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:188)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:387)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format:
"25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:480)
... 15 more
Unfortunately i'm still seeing the same issue. Like your case this issue
comes and goes as well. I am running a 2 node cluster (ES 0.90.7) with
default configuration. I'm using logstash-1.2.2; however looking at the
debug logs within logstash the events are parsed correctly, they just die
when elasticsearch takes the floor.
Sorry I can't be of any more help... If you have any theories, please share
your thoughts.
Cheers,
Ben
On Wednesday, 11 December 2013 21:26:41 UTC+1, Phil Dougherty wrote:
I'm having the same exact issue. Running on a 4 node cluster and using ES
0.90.3. Strangely this issue will randomly bite me, and it seems like it
happens when the index is living on a certain server. Were you able to
resolve this issue or could you provide some more information that may help
me resolve this?
On Monday, November 25, 2013 5:36:41 AM UTC-5, Ben Morrice wrote:
Hello,
I have a basic logstash -> elasticsearch instance running to parse apache
logs; or at least I did. I don't know what has changed but elasticsearch is
now bailing on my timestamp field and is not able to correctly parse it.
I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this
case)
Is anyone able to assist me? I'm sure this is a simple fix?!
Please see below an example from my elasticsearch log
[2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] "GET
/examplerequest HTTP/1.1" 200 11415 "-" "Jakarta
Commons-HttpClient/3.1"","@timestamp":"2013-11-25T08:49:47.548Z","@version":"1","type":"webfrontend","file":"/var/loghttpdssl_access_log","host":" example.com","offset":"629386952","clientip":"10.80.5.30","ident":"-","auth":"-","timestamp":"25/Nov/2013:09:49:45
+0100","verb":"GET","request":"/examplerequest","httpversion":"1.1","response":"200","bytes":"11415","referrer":""-"","agent":""Jakarta
Commons-HttpClient/3.1"","tags":["apache"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:398)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:618)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:471)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:513)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:457)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:342)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed
to parse date field [25/Nov/2013:09:49:45 +0100], tried both date format
[dateOptionalTime], and timestamp number with locale
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:486)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:424)
at
org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:188)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:387)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format:
"25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:480)
... 15 more
judging from a quick peek at the problem: The value you are trying to parse
as the timestamp field (note: not the logstash @timestamp field) is an
unknown date format. This means it will only happen with apache access
logs. Somehow the timestamp field has already been configured to be of a
different format (from a past log entry most likely).
Two possibilites (note: maybe there are others, I am not a logstash wizard)
Make the timestamp field a string and do not analyze it (which also
means, you cannot search for it, but usually you are using the @timestamp
field for that anyway)
Have an own type for the apache access logs in your logstash config,
which uses a special timestamp based mapping format, that fits to CLF
timestamps
Unfortunately i'm still seeing the same issue. Like your case this issue
comes and goes as well. I am running a 2 node cluster (ES 0.90.7) with
default configuration. I'm using logstash-1.2.2; however looking at the
debug logs within logstash the events are parsed correctly, they just die
when elasticsearch takes the floor.
Sorry I can't be of any more help... If you have any theories, please
share your thoughts.
Cheers,
Ben
On Wednesday, 11 December 2013 21:26:41 UTC+1, Phil Dougherty wrote:
I'm having the same exact issue. Running on a 4 node cluster and using ES
0.90.3. Strangely this issue will randomly bite me, and it seems like it
happens when the index is living on a certain server. Were you able to
resolve this issue or could you provide some more information that may help
me resolve this?
On Monday, November 25, 2013 5:36:41 AM UTC-5, Ben Morrice wrote:
Hello,
I have a basic logstash -> elasticsearch instance running to parse
apache logs; or at least I did. I don't know what has changed but
elasticsearch is now bailing on my timestamp field and is not able to
correctly parse it.
I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this
case)
Is anyone able to assist me? I'm sure this is a simple fix?!
Please see below an example from my elasticsearch log
[2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] "GET
/examplerequest HTTP/1.1" 200 11415 "-" "Jakarta
Commons-HttpClient/3.1"","@timestamp":"2013-11-25T08:49:
47.548Z","@version":"1","type":"webfrontend","file":"/var/
loghttpdssl_access_log","host":"example.com","offset":"
629386952","clientip":"10.80.5.30","ident":"-","auth":"-","
timestamp":"25/Nov/2013:09:49:45 +0100","verb":"GET","request":
"/examplerequest","httpversion":"1.1","response":"200","bytes":"11415","
referrer":""-"","agent":""Jakarta Commons-HttpClient/3.1"","
tags":["apache"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:398)
at org.elasticsearch.index.mapper.object.ObjectMapper.
serializeValue(ObjectMapper.java:618)
at org.elasticsearch.index.mapper.object.ObjectMapper.
parse(ObjectMapper.java:471)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:513)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:457)
at org.elasticsearch.index.shard.service.InternalIndexShard.
prepareCreate(InternalIndexShard.java:342)
at org.elasticsearch.action.bulk.TransportShardBulkAction.
shardIndexOperation(TransportShardBulkAction.java:401)
at org.elasticsearch.action.bulk.TransportShardBulkAction.
shardOperationOnPrimary(TransportShardBulkAction.java:155)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction.
performOnPrimary(TransportShardReplicationOperationAction.java:556)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction$1.
run(TransportShardReplicationOperationAction.java:426)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException:
failed to parse date field [25/Nov/2013:09:49:45 +0100], tried both date
format [dateOptionalTime], and timestamp number with locale
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(
DateFieldMapper.java:486)
at org.elasticsearch.index.mapper.core.DateFieldMapper.
innerParseCreateField(DateFieldMapper.java:424)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.
parseCreateField(NumberFieldMapper.java:188)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:387)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format:
"25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.
parseMillis(DateTimeFormatter.java:754)
at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(
DateFieldMapper.java:480)
... 15 more
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.