Need to convert string into date_time format

[vipinkumar]

Hi All

I am working on csv files and in that file there is a column as DATE/TIME in which the date and time is coming in following format.

2017-09-12-0915
YYYY-mm-dd-HHmm

I tried to convert this to DATE/TIME to date_time type by using convert, but in Kibana its type is still string.

Below is my config file.

input {
	beats {
        port => "5043"
    }
}
 filter {
	
	if[fields][document] == "ServerIf"
	{
		csv {
			autodetect_column_names => true
                       separator => ","
			convert => {
				"DATE/TIME" => "date_time"
				"EnquiryIn" => "integer"
				"EnquiryOut" => "integer"
				"BalanceUsed" => "integer"
				"BalanceLeft" => "integer"
			   }
               }
         date {
               match => ["DATE/TIME" , "yyyy-MM-dd HH:mm"]
			   target => "DATE/TIME"
        }
	}
}

output {
           elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

Can anyone help me with this issue...How to convert string to date_time.

[warkolm (Mark Walkom)]

@vipinkumar

convert => {
"DATE/TIME" => "date_time"

That's not valid, there's no type like that.

@vipinkumar

match => ["DATE/TIME" , "yyyy-MM-dd HH:mm"]

That also doesn't match your pattern you originally posted;

@vipinkumar

YYYY-mm-dd-HHmm

[vipinkumar]

Hi Mark

Correct me if I'm wrong, but as per this - https://www.elastic.co/guide/en/elasticsearch/reference/1.7/mapping-date-format.html

I can use date_time instead of date right? In fact originally I tried using date_hour_minute, but it gave this error:

Pipeline aborted due to error {:exception=>#<LogStash::ConfigurationError: Invalid conversion types: date_hour_minute>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-csv-3.0.4/lib/logstash/filters/csv.rb:102:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:281:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:292:in register_plugins'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:292:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:302:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:226:in run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:398:instart_pipeline'"]}

As for your second reply, yes you're right I'm actually trying to change the format YYYY-mm-dd-HHmm to yyyy-MM-dd HH:mm. Bases on your reply I guess this isn't the right approach. Could you kindly guide me.

[magnusbaeck (Magnus Bäck)]

I can use date_time instead of date right?

No. You're referencing the ES documentation which has nothing to do with Logstash.

The date pattern you need is YYYY-MM-dd-HHmm.

[warkolm (Mark Walkom)]

See also https://www.elastic.co/guide/en/logstash/5.5/plugins-filters-date.html#plugins-filters-date-match