Re: having troubles parsing dates

Elastic Stack Elasticsearch
1

Alex,

Thank-you very much for your reply. I've fixed my issue with your help.

For future reference I added the following to my logstash indexer for
apache;

date {
locale => "en"
timezone => "Europe/Zurich"
match => [ "apachetime", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => "@timestamp"
}

Thanks again for your reply.

Cheers,

Ben

On Thu, Dec 12, 2013 at 11:01 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey,

judging from a quick peek at the problem: The value you are trying to
parse as the timestamp field (note: not the logstash @timestamp field) is
an unknown date format. This means it will only happen with apache access
logs. Somehow the timestamp field has already been configured to be of a
different format (from a past log entry most likely).

Two possibilites (note: maybe there are others, I am not a logstash wizard)

  • Make the timestamp field a string and do not analyze it (which also
    means, you cannot search for it, but usually you are using the @timestamp
    field for that anyway)
  • Have an own type for the apache access logs in your logstash config,
    which uses a special timestamp based mapping format, that fits to CLF
    timestamps

See
Elasticsearch Platform — Find real-time answers at scale | Elastic
Elasticsearch Platform — Find real-time answers at scale | Elastic
Elasticsearch Platform — Find real-time answers at scale | Elastic

--Alex

On Thu, Dec 12, 2013 at 10:29 AM, Ben Morrice ben.morrice@gmail.comwrote:

Hi Phil,

Unfortunately i'm still seeing the same issue. Like your case this issue
comes and goes as well. I am running a 2 node cluster (ES 0.90.7) with
default configuration. I'm using logstash-1.2.2; however looking at the
debug logs within logstash the events are parsed correctly, they just die
when elasticsearch takes the floor.

Sorry I can't be of any more help... If you have any theories, please
share your thoughts.

Cheers,

Ben

On Wednesday, 11 December 2013 21:26:41 UTC+1, Phil Dougherty wrote:

I'm having the same exact issue. Running on a 4 node cluster and using
ES 0.90.3. Strangely this issue will randomly bite me, and it seems like it
happens when the index is living on a certain server. Were you able to
resolve this issue or could you provide some more information that may help
me resolve this?

On Monday, November 25, 2013 5:36:41 AM UTC-5, Ben Morrice wrote:

Hello,

I have a basic logstash -> elasticsearch instance running to parse
apache logs; or at least I did. I don't know what has changed but
elasticsearch is now bailing on my timestamp field and is not able to
correctly parse it.

I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this
case)

Is anyone able to assist me? I'm sure this is a simple fix?!

Please see below an example from my elasticsearch log

[2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] "GET
/examplerequest HTTP/1.1" 200 11415 "-" "Jakarta
Commons-HttpClient/3.1"","@timestamp":"2013-11-25T08:49:
47.548Z","@version":"1","type":"webfrontend","file":"/var/
loghttpdssl_access_log","host":"example.com","offset":"
629386952","clientip":"10.80.5.30","ident":"-","auth":"-","
timestamp":"25/Nov/2013:09:49:45 +0100","verb":"GET","request":
"/examplerequest","httpversion":"1.1","response":
"200","bytes":"11415","referrer":""-"","agent":""Jakarta
Commons-HttpClient/3.1"","tags":["apache"]}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
[timestamp]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:398)
at org.elasticsearch.index.mapper.object.ObjectMapper.
serializeValue(ObjectMapper.java:618)
at org.elasticsearch.index.mapper.object.ObjectMapper.
parse(ObjectMapper.java:471)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:513)
at org.elasticsearch.index.mapper.DocumentMapper.parse(
DocumentMapper.java:457)
at org.elasticsearch.index.shard.service.InternalIndexShard.
prepareCreate(InternalIndexShard.java:342)
at org.elasticsearch.action.bulk.TransportShardBulkAction.
shardIndexOperation(TransportShardBulkAction.java:401)
at org.elasticsearch.action.bulk.TransportShardBulkAction.
shardOperationOnPrimary(TransportShardBulkAction.java:155)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction.
performOnPrimary(TransportShardReplicationOperationAction.java:556)
at org.elasticsearch.action.support.replication.
TransportShardReplicationOperationAction$AsyncShardOperationAction$1.
run(TransportShardReplicationOperationAction.java:426)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException:
failed to parse date field [25/Nov/2013:09:49:45 +0100], tried both date
format [dateOptionalTime], and timestamp number with locale
at org.elasticsearch.index.mapper.core.DateFieldMapper.
parseStringValue(DateFieldMapper.java:486)
at org.elasticsearch.index.mapper.core.DateFieldMapper.
innerParseCreateField(DateFieldMapper.java:424)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.
parseCreateField(NumberFieldMapper.java:188)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:387)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format:
"25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.
parseMillis(DateTimeFormatter.java:754)
at org.elasticsearch.index.mapper.core.DateFieldMapper.
parseStringValue(DateFieldMapper.java:480)
... 15 more

Thanks in advance!

Cheers,

Ben

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/087f7e84-261c-4bde-b214-9be3580edae1%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/wUrtAVUE1o4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_eOC8rpT82U_4AQ3GNX93BWvRPZkP9oe1j1q0unRu_BA%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJAhm_oORRy6EJ8YA2r9FBLnjc9%3DEACbu23MNEgvH8BDug1pUw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.