[parsing date error]: date_time_parse_exception: Failed to parse with all enclosed parsers

[Abdelhalim]

Hello,

I am using elasticsearch, Kibana and Logstash all version 7.10.1
I would like to ingest data from URLhaus, so I have created a template like that:

PUT _template/urlhaus
{
  "index_patterns": ["urlhaus-*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "hot-warm-cold-delete-6months-policy",  
    "index.lifecycle.rollover_alias": "urlhaus"
  },
  "mappings": {
    "properties": {
    
     "id": {"type": "long" },
     "dateadded": {"type": "date"},
     "url": { "type": "text"}, 
     "url_status": {"type": "keyword"},
     "threat": {"type": "text"}, 
     "tags": {"type": "text"}, 
     "urlhaus_link": {"type": "text"}, 
     "reporter": {"type": "keyword"}
    }
  } 
}

and and index:

PUT urlhaus-000001
{
  "aliases": {
    "urlhaus": {
      "is_write_index": true
    }
  }
}`

and my logstash configuration :

input {
  
    exec {
      command => 'curl https://urlhaus.abuse.ch/downloads/csv/ --output /etc/logstash/urlhaus/text.zip && unzip -c /etc/logstash/urlhaus/text.zip'
      interval => 86400
      type => 'urlhaus'
      codec => line
    }

}
filter {
  if [type] == "urlhaus" {
    csv {
      columns => ["id","dateadded","url","url_status","threat","tags","urlhaus_link","reporter"]
      separator => ","
    }
    fingerprint {
      concatenate_sources => true
      method => "SHA256"
      source => [ "url" ] # And possibly other fields
      target => "[@metadata][fingerprint]"
    }
    mutate {
      remove_field => ["message"]
    }
  }

}

output {
  if [type] == "urlhaus" {
    elasticsearch {
      hosts => ["https://X.X.X.X:9200"]
      document_id => "%{[@metadata][fingerprint]}"
      cacert => '/etc/logstash/certs/ca.crt'
      user => "elastic"
      password => "password"
      ilm_enabled => auto
      ilm_rollover_alias => "urlhaus"
      ilm_pattern => "000001"
      ilm_policy => "hot-warm-cold-delete-6months-policy"
      template_name => "urlhaus"

    }
  }
}

But I am getting error while trying to parse the field dateadded
an example of the error I am getting:

[WARN ] 2021-01-12 10:30:53.515 [[main]>worker2] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"94e0e042fa24822dd14fcaed4e9b3c90dcc5625f44556acca3b1670c1a5aa102", :_index=>"urlhaus", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x27b1bc7e>], :response=>{"index"=>{"_index"=>"urlhaus-000001", "_type"=>"_doc", "_id"=>"94e0e042fa24822dd14fcaed4e9b3c90dcc5625f44556acca3b1670c1a5aa102", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [dateadded] of type [date] in document with id '94e0e042fa24822dd14fcaed4e9b3c90dcc5625f44556acca3b1670c1a5aa102'. Preview of field's value: '2018-03-05 14:18:19'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2018-03-05 14:18:19] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

An example of the data to index:

"954853","2021-01-12 08:59:04","http://182.117.82.205:37299/i","online","malware_download","32-bit,arm,elf","https://urlhaus.abuse.ch/url/954853/","geenensp"
"954852","2021-01-12 08:58:08","http://182.56.249.80:37275/i","online","malware_download","32-bit,elf,mips","https://urlhaus.abuse.ch/url/954852/","geenensp"
"954851","2021-01-12 08:58:03","http://115.52.34.251:47098/i","online","malware_download","32-bit,elf,mips","https://urlhaus.abuse.ch/url/954851/","geenensp"

Could you tell me please what I am doing wrong ?
Thanks

[Wolfram_Haussig]

Hello Abdelhalim,

According to the docs here:

Date formats can be customised, but if no format is specified then it uses the default:

"strict_date_optional_time||epoch_millis"

strict_date_optional_time is described here:

A generic ISO datetime parser, where the date must include the year at a minimum, and the time (separated by T ), is optional. Examples: yyyy-MM-dd'T'HH:mm:ss.SSSZ or yyyy-MM-dd .

To solve that, you have to update your template and define your date format like this:

PUT _template/urlhaus
{
  "index_patterns": ["urlhaus-*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "hot-warm-cold-delete-6months-policy",  
    "index.lifecycle.rollover_alias": "urlhaus"
  },
  "mappings": {
    "properties": {
    
     "id": {"type": "long" },
     "dateadded": {"type": "date", "format": "yyyy-MM-dd HH:mm:ss"},
     "url": { "type": "text"}, 
     "url_status": {"type": "keyword"},
     "threat": {"type": "text"}, 
     "tags": {"type": "text"}, 
     "urlhaus_link": {"type": "text"}, 
     "reporter": {"type": "keyword"}
    }
  } 
}

Best regards
Wolfram

[Abdelhalim]

Thank you very much @Wolfram_Haussig
I did that, and now it's working like a charm