Logstash date filter error

Elastic Stack Logstash
1

I have this field in my logs event_time=2022-08-30 17:30:42.000

my logstash pipeline file is

filter {
if "average_weight" in [tags] {
kv {
source=>"message"
target => "test"
field_split=>","
}

mutate {
rename => {
"[test][name]" =>"name"
"[test][weight]" => "weight"
"[test][event_time]" => "[event][time]"
}

date {
match =>["[event][time]","YYYY-MM-dd HH:mm:ss.SSS"]
}

remove_tag => ["beats_input_codec_plain_applied"]
convert => {
"weight" => "float"
}

}
}
}

I am getting the error at date filter everthing is fine according to me but i am getting error

2

What is the error? You didn't share any error log or how is your real output.

Please share the error and the output you are getting, also share a sample of your message so it is possible to try to replicate.

3

Try with low letter yyyy

match =>["[event][time]","yyyy-MM-dd HH:mm:ss.SSS"]

Y year of era (>=0) year 1996
y year year 1996

4

Getting the error

[2022-08-31T14:00:28,549][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 25, column 6 (byte 406) after filter {\nif \"average_weight\" in [tags] { \nkv {\nsource=>\"message\"\ntarget => \"test\"\nfield_split=>\",\"\n}\n\nmutate {\nrename => {\n\"[test][name]\" =>\"name\"\n\"[test][weight]\" => \"weight\"\n\"[test][event_time]\" => \"[event][time]\"\n}\n\ndate ", :backtrace=>["/home/aniket/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/home/aniket/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "/home/aniket/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/home/aniket/logstash/logstash-core/lib/logstash/agent.rb:388:in `block in converge_state'"]}
[2022-08-31T14:00:29,797][INFO ][org.reflections.Reflections] Reflections took 307 ms to scan 1 urls, producing 119 keys and 419 values

I've used your date time format

filter {
if "average_weight" in [tags] {
kv {
source=>"message"
target => "test"
field_split=>","
}

mutate {
rename => {
"[test][name]" =>"name"
"[test][weight]" => "weight"
"[test][event_time]" => "[event][time]"
}

date {
match =>["[event][time]","yyyy-MM-dd HH:mm:ss.SSS"]
}

remove_tag => ["beats_input_codec_plain_applied"]
convert => {
"weight" => "float"
}

}
}
}
5

Hi @leandrojmp
I am using beat input where filebeat is sending data to logstash and i want to parse this event_time field as a timestamp
test.log

name=David Kluesner,weight=36,event_time=2022-08-31 14:01:02.066109
name=Pamela Ware,weight=47,event_time=2022-08-31 14:02:01.322111
name=Paul Folsom,weight=40,event_time=2022-08-31 14:02:01.327286
name=Dennis Cochran,weight=34,event_time=2022-08-31 14:02:01.328276
name=Lois Macy,weight=31,event_time=2022-08-31 14:02:01.341071
name=Roberto Kelly,weight=36,event_time=2022-08-31 14:03:01.597030
name=Wesley Ingram,weight=33,event_time=2022-08-31 14:03:01.598002
name=Daniel Kitchen,weight=40,event_time=2022-08-31 14:03:01.601372
name=Julie Obrien,weight=46,event_time=2022-08-31 14:03:01.602028
name=David Taylor,weight=24,event_time=2022-08-31 14:03:01.602174
name=Charles Randle,weight=23,event_time=2022-08-31 14:03:01.604975

I was getting this below error

[2022-08-30T18:05:53,173][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 25, column 6 (byte 406) after filter {\nif \"average_weight\" in [tags] { \nkv {\nsource=>\"message\"\ntarget => \"test\"\nfield_split=>\",\"\n}\n\nmutate {\nrename => {\n\"[test][name]\" =>\"name\"\n\"[test][weight]\" => \"weight\"\n\"[test][event_time]\" => \"[event][time]\"\n}\n\ndate ", :backtrace=>["/home/aniket/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/home/aniket/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "/home/aniket/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/home/aniket/logstash/logstash-core/lib/logstash/agent.rb:388:in `block in converge_state'"]}
6

Add one more }

7

I have used only mutate filter and under this i am using rename,convert,date,remove_tag.

mutate {
rename => {
"[test][name]" =>"name"
"[test][weight]" => "weight"
"[test][event_time]" => "[event][time]"
}

date {
match =>["[event][time]","yyyy-MM-dd HH:mm:ss.SSS"]
}

remove_tag => ["beats_input_codec_plain_applied"]
convert => {
"weight" => "float"
}

}

every curly braces are completed

8

Date is not part of mutate, however, if is working, leave it :blush:

9

Oops i didn't see this in logstash documentation :sweat_smile:

mutate {
rename => {
"[test][name]" =>"name"
"[test][weight]" => "weight"
"[test][event_time]" => "[event][time]"
}

remove_tag => ["beats_input_codec_plain_applied"]
convert => {
"weight" => "float"
}

}

date {
match =>["[event][time]","YYYY-MM-dd HH:mm:ss.SSS"]
}

Its working now

1 Like
10

Just one tip, if you are going to use more than one mutate action in the same field, as destination our source, and need them to happen in a specific order, you should use different mutate blocks.

This is an important not in the documentation:

Each mutation must be in its own code block if the sequence of operations needs to be preserved.

In your case you are doing multiple mutates in the weight field, in your case it worked because rename is processed before convert, but depending on the action the order will change.

1 Like
11

Thank you @leandrojmp i will update my code :blush:

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.