Hello,
I am using elasticsearch, Kibana and Logstash all version 7.10.1
I would like to ingest data from URLhaus, so I have created a template like that:
PUT _template/urlhaus
{
"index_patterns": ["urlhaus-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "hot-warm-cold-delete-6months-policy",
"index.lifecycle.rollover_alias": "urlhaus"
},
"mappings": {
"properties": {
"id": {"type": "long" },
"dateadded": {"type": "date"},
"url": { "type": "text"},
"url_status": {"type": "keyword"},
"threat": {"type": "text"},
"tags": {"type": "text"},
"urlhaus_link": {"type": "text"},
"reporter": {"type": "keyword"}
}
}
}
and and index:
PUT urlhaus-000001
{
"aliases": {
"urlhaus": {
"is_write_index": true
}
}
}`
and my logstash configuration :
input {
exec {
command => 'curl https://urlhaus.abuse.ch/downloads/csv/ --output /etc/logstash/urlhaus/text.zip && unzip -c /etc/logstash/urlhaus/text.zip'
interval => 86400
type => 'urlhaus'
codec => line
}
}
filter {
if [type] == "urlhaus" {
csv {
columns => ["id","dateadded","url","url_status","threat","tags","urlhaus_link","reporter"]
separator => ","
}
fingerprint {
concatenate_sources => true
method => "SHA256"
source => [ "url" ] # And possibly other fields
target => "[@metadata][fingerprint]"
}
mutate {
remove_field => ["message"]
}
}
}
output {
if [type] == "urlhaus" {
elasticsearch {
hosts => ["https://X.X.X.X:9200"]
document_id => "%{[@metadata][fingerprint]}"
cacert => '/etc/logstash/certs/ca.crt'
user => "elastic"
password => "password"
ilm_enabled => auto
ilm_rollover_alias => "urlhaus"
ilm_pattern => "000001"
ilm_policy => "hot-warm-cold-delete-6months-policy"
template_name => "urlhaus"
}
}
}
But I am getting error while trying to parse the field dateadded
an example of the error I am getting:
[WARN ] 2021-01-12 10:30:53.515 [[main]>worker2] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"94e0e042fa24822dd14fcaed4e9b3c90dcc5625f44556acca3b1670c1a5aa102", :_index=>"urlhaus", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x27b1bc7e>], :response=>{"index"=>{"_index"=>"urlhaus-000001", "_type"=>"_doc", "_id"=>"94e0e042fa24822dd14fcaed4e9b3c90dcc5625f44556acca3b1670c1a5aa102", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [dateadded] of type [date] in document with id '94e0e042fa24822dd14fcaed4e9b3c90dcc5625f44556acca3b1670c1a5aa102'. Preview of field's value: '2018-03-05 14:18:19'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2018-03-05 14:18:19] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}
An example of the data to index:
"954853","2021-01-12 08:59:04","http://182.117.82.205:37299/i","online","malware_download","32-bit,arm,elf","https://urlhaus.abuse.ch/url/954853/","geenensp"
"954852","2021-01-12 08:58:08","http://182.56.249.80:37275/i","online","malware_download","32-bit,elf,mips","https://urlhaus.abuse.ch/url/954852/","geenensp"
"954851","2021-01-12 08:58:03","http://115.52.34.251:47098/i","online","malware_download","32-bit,elf,mips","https://urlhaus.abuse.ch/url/954851/","geenensp"
Could you tell me please what I am doing wrong ?
Thanks