Couldn't convert date_time type

I want to convert the column from string to date_time but nothing changed.

log

2018-04-13T05:37:52.874Z|8|200|0|USR|1510|[Fire...|104|2018-04-12T11:05:15.352Z|201|Pacific|1510|[Fire...|R110|2018-04-12T11:05:59.352Z

configuration(filter part)

filter {
	if "mongo" in [tags] {
		csv{
			separator => "|"
			columns => [
				"logTime",
				"transactionId",
				"resultCode",
				"reqId",
				"reqType",
				"reqCode",
				"reqParam",
				"reqBytes",
				"reqTime",
				"resId",
				"resType",
				"resCode",
				"resParam",
				"resBytes",
				"resTime"
			]
			convert => {
				"logTime" => "date_time"
				"reqTime" => "date_time"
				"resTime" => "date_time"
			}  
			remove_field => ["@version","host","path"]
		}
	}
}

10

I presume the output you show is from a Mongo data explorer of some kind.
I presume that you are using the mongodb output. You should really give as much detail as possible.

I don't think you can assume that the CSV filter convert are failing, the mongo output converts LogStash::Timestamp to String by default.
From the comments in the code..

  # If true, store the @timestamp field in MongoDB as an ISODate type instead
  # of an ISO8601 string.  For more information about this, see
  # http://www.mongodb.org/display/DOCS/Dates.
  config :isodate, :validate => :boolean, :default => false

A better link is https://docs.mongodb.com/manual/reference/bson-types/index.html#date

Our docs on the isodate setting.

I've solved the problem.
thank you

filter {
	if "mongo" in [tags] {
		csv{
			separator => "|"
			columns => [
				"logTime",
                                .
                                .
 				"reqTime",
                                .
                                .
				"resTime"
			]
			convert => {
				"logTime" => "date_time"
				"reqTime" => "date_time"
				"resTime" => "date_time"
			}  
			remove_field => ["@version","host","path"]
		}
        date{
        	match => ["logTime", "ISO8601" ]
			target => "logTime"
        }
        date{
        	match => ["reqTime", "ISO8601" ]
			target => "reqTime"
        }
        date{
        	match => ["resTime", "ISO8601" ]
			target => "resTime"
        }
	}
}

output {  
	stdout {
    		codec => rubydebug
  	}
	if "mongo" in [tags] {
		mongodb { 
			collection => "ServiceLog" 
			database => "log"
			isodate => true
			uri => "mongodb://localhost"
		}
	}
}
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.