"Invalid format" message for a date format that always worked. Why the sudden date parsing error?

Elastic Stack Logstash
1

Hello,
I have had csv files for a long time parsing correctly, but something changed and I get the error below.
My "run_date" is a simple field that looks like this in the data files: "2017-12-28"

It always worked. I mutated it like this, never had a problem.

date {
match => [ "run_date", "yyyy-MM-dd" ]
}

My data csv files changed, got added columns, but nothing relevant to this problem has changed.
The main change is the upgrade to ELK 6.X but maybe it's just a coincidence.

Below is the error log related to this post, and I cannot understand the problem from the log.
Any advice greatly appreciated.

[2018-01-10T16:17:22,977][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"c6f45790ab274ac98083dec77711f6d9f8d19496", :_index=>"company_test", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x4e24f933>], :response=>{"index"=>{"_index"=>"company_test", "_type"=>"doc", "_id"=>"c6f45790ab274ac98083dec77711f6d9f8d19496", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [run_date]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"run_date\""}}}}}

Best,
Dor Juravski

2

Hi @dorj1234,

try this one i hope its working.

date 
{
	match => ["run_date", "UNIX_MS"]
	target => "run_date_New"
}
3

Thank you for the suggestion Krinal - UNIX_MS is meant to parse int values (see here) and I have a yyyy-MM-dd string to parse.

Any other suggestions by anyone? thank you in advance.
JD

4

It looks like you're trying to index a document where the run_date field contains the string "run_date". Perhaps a column header?

5

I wish it was that, but:

  1. I always had a conf line that drops the header. The drop runs before the date clause.
  2. It always worked, and my conf file never changed, my data never changed

The only thing that changed to my knowledge is the logstash version.

If I am wrong in the above assumptions - can you suggest a debug option that will show the value of run_date and information on the specific csv file with the bad input?

6

If I am wrong in the above assumptions - can you suggest a debug option that will show the value of run_date and information on the specific csv file with the bad input?

You could e.g. temporarily replace the elasticsearch output with a stdout { codec => elasticsearch } output so that Logstash dumps the raw events into its log.

1 Like
7

I did that, it looked as expected. yyyy-MM-dd.
I am guessing that the records that raise this error don't show up on stdout, do they?
My goal is to find them, maybe understand if there is a difference.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.