Unable to convert fields into DATE type in Logstash

Elastic Stack Logstash
1

Hello team,
I am unbale to convert field into data type. Can you please help me on this. Format is coming as Text only

Sample Log:

CIG,CCI05_05_NRW_22102020043728_000014635.txt,26/07/2022 04:37:50,26/07/2022 04:42:12,CCI05_05_NRW_22102020043728_000014635.txt,26-07-2022 03:00,26-07-2022 04:42,CCI05_05_NRW_22102020043728_000014635.txt,04:42:33,,,SENT_FROM_BPM,,,,,

GROK:

%{GREEDYDATA},\s*%{DATE:date1}\s*%{TIME:time1}\,\s*%{DATE:date2}\s*%{TIME:time2}

Logstash Conf:

input {
  beats {
    port => 5044
  }
}

filter {
	grok {
	 match => { "message" => "%{GREEDYDATA},\s*%{DATE:date1}\s*%{TIME:time1}\,\s*%{DATE:date2}\s*%{TIME:time2}"
	 }
	 }
		mutate {
			add_field => {
				"Source_Creation_Date" => "%{date1} %{time1}"
				"Source_updation_Date" => "%{date2} %{time2}"
			}
		}
			date {
				match => [ "Source_Creation_Date", "dd/MM/yyyy HH:mm:ss", "dd/MM/yyyy  HH:mm:ss" ]
      }
	  date {
				match => [ "Source_updation_Date", "dd/MM/yyyy HH:mm:ss", "dd/MM/yyyy  HH:mm:ss" ]
      }
    }

output {
  elasticsearch {
   hosts => ["http://localhost:9200"]
   index => "testindexdata6"
  }
  stdout { codec => rubydebug }
}
2

Hi @mangeshmj1992
date filter in Logstash is used to override @timestamp field with other value to your choice.
If you want Elasticseach to treat Source_Creation_Date and Source_updation_Date as dates then you can set the appropriate mapping when creating the index (using index template or manually).
e.g. manually (using kibana dev tools - if index exists delete it first as mapping cannot be changed):
PUT testindexdata6

{
  "mappings": {
    "properties": {
      "Source_Creation_Date": {
        "type": "date",
        "format": "dd/MM/yyyy HH:mm:ss"
      },
      "Source_updation_Date": {
        "type": "date",
        "format": "dd/MM/yyyy HH:mm:ss"
      }
    }
  }
}
  • You can remove date filters from Logstash pipeline
  • Bear in mind that you are not using timezone which can be fine if your data is from same timezone but is less recommended. Elasticsearch & Kibana "now" operator is always compared to UTC time.
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.