Date Parse error Invalid format:


(Ashwin Vemula) #1

curl -XPUT "http://localhost:9200/citrix" -H 'Content-Type: application/json' -d'
{
"mappings" : {
"session_details" : {
"properties" : {
"ASSOCIATED_USER" : {"type": "text" },
"MACHINE_NAME" : {"type": "text" },
"DELIVERY_GROUP" : { "type" : "keyword" },
"SESSION_START_TIME" : { "type" : "date" },
"SESSION_END_TIME" : { "type" : "date" },
"SESSION_DURATION" : { "type" : "text" }
}
}
}
}'

I am facing an error, here is message
[WARN ] 2018-06-26 15:13:29.039 [Ruby-0-Thread-8@[main]>worker3: :1] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"citrix", :_type=>"session_details", :_routing=>nil}, #LogStash::Event:0x9515996], :response=>{"index"=>{"_index"=>"citrix", "_type"=>"session_details", "_id"=>"dUKDPWQBRAHJjbi02-KY", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [SESSION_START_TIME]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "6/7/2018 9:56:41 AM" is malformed at "/7/2018 9:56:41 AM""}}}}}

data: csv file

A.Sivasan,CTXPW021,Apps,6/07/2018 4:40:22 PM,6/7/2018 4:41:06 PM,0:0
P.Kumar,CTXPW128,Apps,6/8/2018 3:10:07 AM,6/8/2018 11:11:23 AM,8:1
tjenning.h,CTXPW317,Desktop,6/5/2018 7:41:59 AM,6/6/2018 12:58:54 AM,17:16
jerome.huang,CTXPW255,Desktop,6/7/2018 11:16:09 AM,6/8/2018 12:33:16 AM,13:17
shaik.rasool,CTXPW356,Desktop,6/7/2018 12:08:55 AM,6/7/2018 1:38:07 AM,1:29
P.Kodandak,CTXPW185,Desktop 2013,6/8/2018 3:27:30 AM,6/8/2018 9:38:18 PM,18:10
annetria.holty,CTXPW493,Desktop 2013,6/4/2018 10:46:49 AM,6/4/2018 4:58:39 PM,6:11

Logstash config file
input {
file{
path => "Details.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["ASSOCIATED_USER","MACHINE_NAME","DELIVERY_GROUP","SESSION_START_TIME","SESSION_END_TIME","SESSION_DURATION"]
}
date {
match => [ "SESSION_START_TIME","d/M/YYYY hh:mm:ss aa","dd/MM/YYYY hh:mm:ss aa"]
}
}
output {
elasticsearch {
hosts => "localhost"
index => "citrix"
document_type => "session_details"
}
stdout {}
}

could someone please help on date parse error.

Thank you
Ashwin Vemula


(Magnus Bäck) #2

If you want the date filter to store the parsed timestamp back into the SESSION_START_TIME field you need to use its target option.


(Ashwin Vemula) #3

Thank you for your reply, but even after adding target field to SESSION_START_TIME till I am getting same error.

Error Message:

[2018-06-27T09:45:08,119][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"citrix_argo", :_type=>"session_details", :_routing=>nil}, #LogStash::Event:0x1e77340], :response=>{"index"=>{"_index"=>"citrix_argo", "_type"=>"session_details", "_id"=>"fqZ9QWQBDfOrb0fwmiWq", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [SESSION_START_TIME]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "6/8/2018 3:07:36 PM" is malformed at "/8/2018 3:07:36 PM""}}}}}

Logstash config:

input {
file{
path => "C:\Users\Ashwin.Vemula\Documents\Task\InsightProject\Citrix\Argo_Details_test.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["ASSOCIATED_USER","MACHINE_NAME","DELIVERY_GROUP","SESSION_START_TIME","SESSION_END_TIME","SESSION_DURATION"]
}
date {
match => [ "SESSION_START_TIME", "d/M/yyyy hh:mm:ss a", "d/M/yyyy hh:mm:ss aa", "d/M/yyyy hh:mm:ss aa", "dd/MM/yyyy hh:mm:ss aa", "d/M/YYYY hh:mm:ss a", "d/M/YYYY hh:mm:ss aa", "d/M/YYYY hh:mm:ss aa", "dd/MM/YYYY hh:mm:ss aa" ]
target => "SESSION_START_TIME"
}

}
output {
elasticsearch {
hosts => "localhost"
index => "citrix_argo"
document_type => "session_details"
}
stdout {}
}

With regards,
Ashwin Vemula


(Magnus Bäck) #4

This indicates that the date filter is failing. Comment out your elasticsearch output and use a stdout { codec => rubydebug } output to verify what you're actually trying to send to ES.


(Ashwin Vemula) #5

here is output, its sending to Elastic-search

LogStash Error Message:

[2018-06-27T10:01:10,582][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"citrix_argo", :_type=>"session_details", :_routing=>nil}, #LogStash::Event:0x640552], :response=>{"index"=>{"_index"=>"citrix_argo", "_type"=>"session_details", "_id"=>"kKaMQWQBDfOrb0fwSiVY", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [SESSION_START_TIME]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "6/8/2018 8:37:50 AM" is malformed at "/8/2018 8:37:50 AM""}}}}}
{
"SESSION_DURATION" => "0:0",
"DELIVERY_GROUP" => "Apps",
"SESSION_START_TIME" => "6/7/2018 4:40:22 PM",
"message" => "ASivasan,CTXPW001, Apps,6/7/2018 4:40:22 PM,6/7/2018 4:41:06 PM,0:0\r",
"MACHINE_NAME" => "CTXPW001",
"path" => "C:\Users\Ashwin.Vemula\Documents\Task\InsightProject\Citrix\Details_test.csv",
"host" => "S24-15TJG22",
"tags" => [
[0] "_dateparsefailure"
],
"ASSOCIATED_USER" => "ASivasan",
"@version" => "1",
"SESSION_END_TIME" => "6/7/2018 4:41:06 PM",
"@timestamp" => 2018-06-27T14:01:09.921Z
}

Elasticsearch Error message:

[2018-06-27T10:01:10,535][DEBUG][o.e.a.b.TransportShardBulkAction] [citrix_argo][2] failed to execute bulk item (index) BulkShardRequest [[citrix_argo][2]] containing [2] requests
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [SESSION_START_TIME]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:302) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:607) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:407) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:93) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:67) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:261) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:714) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:692) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:673) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequestOnPrimary(TransportShardBulkAction.java:548) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeIndexRequest(TransportShardBulkAction.java:140) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:236) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:123) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:110) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:72) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1034) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1012) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:103) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:359) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:299) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:975) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:972) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:238) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2221) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:984) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:98) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:320) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:295) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:282) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656) [elasticsearch-6.2.4.jar:6.2.4]

With regards
Ashwin Vemula


(Magnus Bäck) #6

Look at what the date filter is logging instead. It'll tell you why it can't parse your timestamp.


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.