[Resolved]Logstash - Question for how to use if condition to skip invalid date format

[cheriemilk]

Hi Team,

I have a CSV file which has a column named 'COMMIT_DATE', the values of this column input by end user sometimes are a string , instead of a date. For example, there's invalid date value 'test' in below source data.

GIT_ORG,GIT_REPOS,COMMIT_SHA1,COMMIT_AUTHOR,COMMIT_DATE
bizx,au-recruiting,6739c82bcf830b05d4d36e9fd715bc5715e0c380,Kaderjan Ilghar,2018-01-24
bizx,au-V4,72db50e50121110e98ccb18d90b47f227df96ea1,csong,test
bizx,idl-analytics-api,1be44f52f25f6b540f284eb17e8cee5838826cb9,ssheriff,2017-03-30
bizx,au-employeecentral,23d7a081034c2a27b43686ab9a3468774b03e07d,Felix Roth,test

I want logstash not process and skip the records that contains invalid date format with applying the if condition. Here is the logstash.conf but it doesn't work as expected. Can you help how to write the right if condition?

filter { 
        if [COMMIT_DATE] == "yyyy-MM-dd" {
        csv { columns => [ "GIT_ORG",
                            "GIT_REPOS",
                            "COMMIT_SHA1",
                            "COMMIT_AUTHOR",
							"COMMIT_DATE" ]
               separator => ","
			   skip_header => "true"
			   }
		}

[Badger]

I would use a date filter to parse COMMIT_DATE and then

if _dateparsefailure in [tags] { drop {} }

[cheriemilk]

Badger -Error is returned with running below configuration, doesn't understand what format error it means.

[2019-04-01T10:27:32,953][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])\"{\n  \"template\": \"test1\",\n
  \"index_patterns\":[\"test1\"],\n  \"settings\": {\n    \"number_of_shards\": 1,\n\t\"number_of_replicas\" : 1,\n\t\"refresh_interval\": \"5s\" \n  },\n\n  \"mappings\": {\n    \"doc\": {\n      \"dynamic_templates\": \n      {\
n          \"match\": \"*\",\n          \"match_mapping_type\": \"*\",\n          \"mapping\": {\n            \"type\": \"keyword\"\n          }\n      }\n    }\n  }\n\n\"; line: 1, column: 1])\n at [Source: (byte[])\"{\n  \"templ
ate\": \"test1\",\n  \"index_patterns\":[\"test1\"],\n  \"settings\": {\n    \"number_of_shards\": 1,\n\t\"number_of_replicas\" : 1,\n\t\"refresh_interval\": \"5s\" \n  },\n\n  \"mappings\": {\n    \"doc\": {\n      \"dynamic_temp
lates\": \n      {\n          \"match\": \"*\",\n          \"match_mapping_type\": \"*\",\n          \"mapping\": {\n            \"type\": \"keyword\"\n          }\n      }\n    }\n  }\n\n\"; line: 23, column: 369]", :class=>"LogS
tash::Json::ParserError", :backtrace=>["C:/elkstack/logstash-6.5.1/logstash-6.5.1/logstash-core/lib/logstash/json.rb:16:in `jruby_load'", "C:/elkstack/logstash-6.5.1/logstash-6.5.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-el
asticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:33:in `read_template_file'", "C:/elkstack/logstash-6.5.1/logstash-6.5.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logs
tash/outputs/elasticsearch/template_manager.rb:17:in `get_template'", "C:/elkstack/logstash-6.5.1/logstash-6.5.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/template_m
anager.rb:7:in `install_template'", "C:/elkstack/logstash-6.5.1/logstash-6.5.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:118:in `install_template'", "C:/el
kstack/logstash-6.5.1/logstash-6.5.1/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.1-java/lib/logstash/outputs/elasticsearch/common.rb:49:in `block in install_template_after_successful_connection'"]}
[2019-04-01T10:27:33,519][TRACE][logstash.inputs.file     ] Registering file input {:path=>["C:/elkstack/elasticsearch-6.5.1/logs/test1.csv"]}

logstash configuraiton

filter { date { match => [ "COMMIT_DATE", "yyyy-MM-dd" ]
                target => "COMMIT_DATE" }
                   
        if "_dateparsefailure" in [tags] { drop {} }

        csv { columns => [ "GIT_ORG",
                            "GIT_REPOS",
                            "COMMIT_SHA1",
                            "COMMIT_AUTHOR",
							"COMMIT_DATE" ]
               separator => ","
			   skip_header => "true"
			   }
		}

[cheriemilk]

I figure out the error is in index template. Thanks