CSV _dateparsefailure error

Elastic Stack Logstash
1

Not sure whats going on but I keep getting a dateparsefailure tag on all my data that comes in. Ive tried this on two different ELk instances and I still get the tag attached to my data. Any idea whats going on? I have a sample entry from the CSV below as well as the filter for this data.

Sample Entry:
wannacry,5b38202756b09192cfbdf758c5540579,7d0a15034243f3eb3748b011c2a293e0c436d4e2,a6af49137e4924ee3af1a16ff004e7b9e1fbe1c32adb4797d98e07e6088b16e0,52,Win32 DLL,5384861005840384,2019-02-27 21:19:58

Snippet of filter from config file:
...
if "rss" in [tags] {
csv {
columns => ["family","md5","sha1","sha256","positives","file_type","id","date"]
}
date {
match => ["date", "yyyy MM dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss"]
timezone => "UTC"
}
}
...

2

Are you sure that is the only date filter in the configuration?

3

On the PROD instance, there are two date filters but they are contained in if statements for different indexes shown below:

filter {
if "hashes" in [tags] {
csv {
columns => ["tag", "date", "hash"]
}
date {
match => ["date", "yyyy MMM dd HH:mm:ss"]
timezone => "America/Chicago"
}
}
if "rss" in [tags] {
csv {
columns => ["family","md5","sha1","sha256","positives","file_type","id","date"]
}
date {
match => ["date", "yyyy MM dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss"]
timezone => "UTC"
}
}
}

The top "if" works fine but the bottom doesn't work at all and I'm really not sure why. Do I need to chance "MM" to "MMM"? I've tried echoing the date into logstash via the command line with my conf file but that just keeps erroring out.

4

Does an event ever have both tags?

5

They shouldn't, I assign the tags depending on which csv the data comes from. Input and output are shown below:
Input:
input {
file {
path => "/dir/dir/dir/dir/trendsv2.csv"
start_position => "beginning"
sincedb_path => "/dev/null/"
tags => 'hashes'
}

	file {
                path => "/dir/dir/dir/dir/rss_feed/trendsv3.csv"
                start_position => "beginning"
                sincedb_path => "/dev/null/"
				tags => 'rss'
        }
}

Output:
output {
if "hashes" in [tags] {
elasticsearch {
hosts => "localhost:9200"
index => "hashes"
document_type => "malware_trends"
}
}
if "rss" in [tags] {
elasticsearch {
hosts => "localhost:9200"
index => "rss_feed"
document_type => "malware_trends"
}
}
stdout {}
}

6

Can you show a sample event that gets written to stdout with a _dateparsefailure tag?

7

This is what I am seeing in my syslog. I expect the second message since it is the header of the csv file. Ill get them from the PROD instance here in a bit.

Mar  4 10:56:16 ubuntu logstash[6201]:[1] "_dateparsefailure"

Mar  4 11:20:43 ubuntu logstash[6201]: [2019-03-04T11:20:43,082][DEBUG][logstash.pipeline] output received {"event"=>{"file_type"=>"type", "@timestamp"=>2019-03-04T19:20:42.263Z, "sha1"=>"sha1", "family"=>"tag", "md5"=>"md5", "host"=>"ubuntu", "date"=>"date", "sha256"=>"sha256", "path"=>"/home/dev/trends/trendsv3.csv", "message"=>"tag,md5,sha1,sha256,positives,type,id,date\r", "tags"=>["rss", "_dateparsefailure"], "@version"=>"1", "id"=>"id", "positives"=>"positives"}}

Mar  4 11:20:44 ubuntu logstash[6201]:[1] "_dateparsefailure"

Mar  4 10:56:06 ubuntu logstash[6201]: [2019-03-04T10:56:06,025][DEBUG][logstash.filters.date] config LogStash::Filters::Date/@tag_on_failure = ["_dateparsfailure"]
8

For the header row I would expect to get the _dateparsefailure.

9

Well Badger, it looks like everything is working fine now. I really have no clue what was going on. These were the last dateparsefailure errors I received in the syslog on the PROD instance when I copied the conf file from the test instance.

Mar  4 14:49:10 PRODBOX logstash[13570]: [2019-03-04T14:49:10,247][DEBUG][logstash.pipeline        ] output received {"event"=>{"message"=>"family,md5,sha1,sha256,positives,type,id,date\r", "sha256"=>"sha256", "path"=>"/dir/dir/dir/dir/trendsv3.csv", "@timestamp"=>2019-03-04T20:49:09.096Z, "md5"=>"md5", "host"=>"PRODBOX", "@version"=>"1", "family"=>"family", "date"=>"date", "tags"=>["rss", "_dateparsefailure"], "sha1"=>"sha1", "id"=>"id", "positives"=>"positives", "file_type"=>"type"}}
Mar  4 14:49:11 PRODBOX logstash[13570]:         [1] "_dateparsefailure"
Mar  4 14:49:12 PRODBOX logstash[13570]:         [1] "_dateparsefailure"
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.