Date Parse error Invalid format:

curl -XPUT "http://localhost:9200/citrix" -H 'Content-Type: application/json' -d'
{
"mappings" : {
"session_details" : {
"properties" : {
"ASSOCIATED_USER" : {"type": "text" },
"MACHINE_NAME" : {"type": "text" },
"DELIVERY_GROUP" : { "type" : "keyword" },
"SESSION_START_TIME" : { "type" : "date" },
"SESSION_END_TIME" : { "type" : "date" },
"SESSION_DURATION" : { "type" : "text" }
}
}
}
}'

I am facing an error, here is message
[WARN ] 2018-06-26 15:13:29.039 [Ruby-0-Thread-8@[main]>worker3: :1] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"citrix", :_type=>"session_details", :_routing=>nil}, #LogStash::Event:0x9515996], :response=>{"index"=>{"_index"=>"citrix", "_type"=>"session_details", "_id"=>"dUKDPWQBRAHJjbi02-KY", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [SESSION_START_TIME]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "6/7/2018 9:56:41 AM" is malformed at "/7/2018 9:56:41 AM""}}}}}

data: csv file

A.Sivasan,CTXPW021,Apps,6/07/2018 4:40:22 PM,6/7/2018 4:41:06 PM,0:0
P.Kumar,CTXPW128,Apps,6/8/2018 3:10:07 AM,6/8/2018 11:11:23 AM,8:1
tjenning.h,CTXPW317,Desktop,6/5/2018 7:41:59 AM,6/6/2018 12:58:54 AM,17:16
jerome.huang,CTXPW255,Desktop,6/7/2018 11:16:09 AM,6/8/2018 12:33:16 AM,13:17
shaik.rasool,CTXPW356,Desktop,6/7/2018 12:08:55 AM,6/7/2018 1:38:07 AM,1:29
P.Kodandak,CTXPW185,Desktop 2013,6/8/2018 3:27:30 AM,6/8/2018 9:38:18 PM,18:10
annetria.holty,CTXPW493,Desktop 2013,6/4/2018 10:46:49 AM,6/4/2018 4:58:39 PM,6:11

Logstash config file
input {
file{
path => "Details.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["ASSOCIATED_USER","MACHINE_NAME","DELIVERY_GROUP","SESSION_START_TIME","SESSION_END_TIME","SESSION_DURATION"]
}
date {
match => [ "SESSION_START_TIME","d/M/YYYY hh:mm:ss aa","dd/MM/YYYY hh:mm:ss aa"]
}
}
output {
elasticsearch {
hosts => "localhost"
index => "citrix"
document_type => "session_details"
}
stdout {}
}

could someone please help on date parse error.

Thank you
Ashwin Vemula

If you want the date filter to store the parsed timestamp back into the SESSION_START_TIME field you need to use its target option.

Thank you for your reply, but even after adding target field to SESSION_START_TIME till I am getting same error.

Error Message:

[2018-06-27T09:45:08,119][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"citrix_argo", :_type=>"session_details", :_routing=>nil}, #LogStash::Event:0x1e77340], :response=>{"index"=>{"_index"=>"citrix_argo", "_type"=>"session_details", "_id"=>"fqZ9QWQBDfOrb0fwmiWq", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [SESSION_START_TIME]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "6/8/2018 3:07:36 PM" is malformed at "/8/2018 3:07:36 PM""}}}}}

Logstash config:

input {
file{
path => "C:\Users\Ashwin.Vemula\Documents\Task\InsightProject\Citrix\Argo_Details_test.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["ASSOCIATED_USER","MACHINE_NAME","DELIVERY_GROUP","SESSION_START_TIME","SESSION_END_TIME","SESSION_DURATION"]
}
date {
match => [ "SESSION_START_TIME", "d/M/yyyy hh:mm:ss a", "d/M/yyyy hh:mm:ss aa", "d/M/yyyy hh:mm:ss aa", "dd/MM/yyyy hh:mm:ss aa", "d/M/YYYY hh:mm:ss a", "d/M/YYYY hh:mm:ss aa", "d/M/YYYY hh:mm:ss aa", "dd/MM/YYYY hh:mm:ss aa" ]
target => "SESSION_START_TIME"
}

}
output {
elasticsearch {
hosts => "localhost"
index => "citrix_argo"
document_type => "session_details"
}
stdout {}
}

With regards,
Ashwin Vemula

This indicates that the date filter is failing. Comment out your elasticsearch output and use a stdout { codec => rubydebug } output to verify what you're actually trying to send to ES.

here is output, its sending to Elastic-search

LogStash Error Message:

[2018-06-27T10:01:10,582][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"citrix_argo", :_type=>"session_details", :_routing=>nil}, #LogStash::Event:0x640552], :response=>{"index"=>{"_index"=>"citrix_argo", "_type"=>"session_details", "_id"=>"kKaMQWQBDfOrb0fwSiVY", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [SESSION_START_TIME]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "6/8/2018 8:37:50 AM" is malformed at "/8/2018 8:37:50 AM""}}}}}
{
"SESSION_DURATION" => "0:0",
"DELIVERY_GROUP" => "Apps",
"SESSION_START_TIME" => "6/7/2018 4:40:22 PM",
"message" => "ASivasan,CTXPW001, Apps,6/7/2018 4:40:22 PM,6/7/2018 4:41:06 PM,0:0\r",
"MACHINE_NAME" => "CTXPW001",
"path" => "C:\Users\Ashwin.Vemula\Documents\Task\InsightProject\Citrix\Details_test.csv",
"host" => "S24-15TJG22",
"tags" => [
[0] "_dateparsefailure"
],
"ASSOCIATED_USER" => "ASivasan",
"@version" => "1",
"SESSION_END_TIME" => "6/7/2018 4:41:06 PM",
"@timestamp" => 2018-06-27T14:01:09.921Z
}

Elasticsearch Error message:

[2018-06-27T10:01:10,535][DEBUG][o.e.a.b.TransportShardBulkAction] [citrix_argo][2] failed to execute bulk item (index) BulkShardRequest [[citrix_argo][2]] containing [2] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [SESSION_START_TIME]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:607) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:407) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:67) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:261) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:714) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:692) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:673) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequest(TransportShardBulkAction.java:140) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:236) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1034) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1012) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:103) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:359) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:299) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:975) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:972) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2221) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:984) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:98) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:320) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:295) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:282) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656) [elasticsearch-6.2.4.jar:6.2.4]

With regards
Ashwin Vemula

Look at what the date filter is logging instead. It'll tell you why it can't parse your timestamp.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.