Convert string in strict_date_time (yyyy-MM-dd'T'HH:mm:ss.SSSZZ) to @timestamp

Elastic Stack Logstash
1

Hi, I'am trying to convert a string in strict_date_time format 2021-02-19T23:02:57.277+05:30 to @timestamp.

date {
match => [ "timeStampString" , "yyyy-MM-dd'T'HH:mm:ss.SSSZZ" ]
target => "logTimestamp3"
}

Throwing "_dateparsefailure" while trying to convert. Need to know how to conver the string 2021-02-19T23:02:57.277+05:30 to @timestamp

2

Think just using ISO8601 should work for this one.

  date {
    match => [ "timeStampString" , "ISO8601" ]
    target => "logTimestamp3"
  }
3

I have tried this already and it is throwing _dateparse exception.

4

I tested using this. Is there something different about what you are doing?

Conf

input {
  generator {
    lines => [ '{"timeStampString": "2021-02-19T23:02:57.277+05:30"}']
    count => 1
    codec => "json"
  }
}
filter {
  date {
    match => [ "timeStampString" , "ISO8601" ]
    target => "logTimestamp3"
  }
}
output {  stdout { codec =>  "rubydebug" } }

Output

{
    "timeStampString" => "2021-02-19T23:02:57.277+05:30",
      "logTimestamp3" => 2021-02-19T17:32:57.277Z,
}
5

This did not work for me. I'am putting my file config, input and output below

7

Config file below

input {
  file {
    path => "C:/zz_mtemp/elk/inp/abc.txt"
	start_position => "beginning"
	delimiter => "\r\n"	
  }
}
filter
{
  xml {
	remove_namespaces => "true"
	source => "message"
	store_xml => "false"
	target => "doc"
	xpath =>  [
	  "Logger_Request/conversationId/text()","conversationId",
	  "Logger_Request/logTimestamp/text()","timeStampString"	  
    ]
  }
 
  mutate {
    remove_field => [ "message" ]
	remove_field => [ "path" ]
	remove_field => [ "@timestamp" ]
	remove_field => [ "@version" ]
	remove_field => [ "host" ]
  }
  date {
    match => [ "timeStampString" , "ISO8601" ]	
	target => "logTimestamp3"
  }
  
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
	index => "filexmlreadindex"	
  }
  stdout { codec => rubydebug }
}
8

Input data in file

<tns2:Logger_Request xmlns:tns="http://www.tibco.com/namespaces/tnt/plugins/jms+32553768-9d2e-4ed0-90ae-0a1e20803547+input" xmlns:tns2="http://www.ericsson.com/tibco/schema/Logger" xmlns:xml="http://www.w3.org/XML/1998/namespace"><tns2:conversationId>TIB-d2a12b2b-785a-4f92-905d-69132363e001</tns2:conversationId><tns2:logTimestamp>2021-02-19T23:02:57.277+05:30</tns2:logTimestamp></tns2:Logger_Request>
9

Output in command line

[2021-02-24T20:59:33,450][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
{
    "timeStampString" => [
        [0] "2021-02-19T23:02:57.277+05:30"
    ],
               "tags" => [
        [0] "_dateparsefailure"
    ],
     "conversationId" => [
        [0] "TIB-d2a12b2b-785a-4f92-905d-69132363e001"
    ]
}
10

timeStampString is an array. Try

match => [ "[timeStampString][0]" , "ISO8601" ]

or else add force_array => false to your xml filter.

11 
  date {
    match => [ "[timeStampString][0]" , "ISO8601" ]	
    target => "logTimestamp3"
  }
12

Thankyou.. This solution worked.

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.