Hi all
I am using filebeat 7.8 which sends log to logstash 7.8. Each line of log is as following:
{"req": { "service": ser1,"re_date": "2020-09-10T09:14:38.479"}}
and my logstash filter is as following:
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["0.0.0.0:9200"]
index => "test"
}
stdout { codec => rubydebug }
}
which will be index in elasticsearch as following event:
req.service: ser1
req.re_date: 2020-09-10T09:14:38.479
Now, i want to change req.re_date format to (2020/09/10T09:14:38.479) using following filter, but the output is still same and dose not make any change.
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
mutate {
gsub => [
"%{req.re_date}", "-", "/"
]
}
}
output {
elasticsearch {
hosts => ["0.0.0.0:9200"]
index => "test"
}
stdout { codec => rubydebug }
}
Any advise will be so appreciated.
Regards
Try gsub => [ "[req][re_date]", "-", "/" ]
1 Like
Many thanks for your reply. But, output is still the same as "2020-09-10T09:14:38.479". Following warning has been found in the log:
[2021-04-24T17:34:28,262][WARN ][logstash.filters.mutate ][main][e1ebc65d12b5ed0ee94689d88d19899b9e56d1373846c87a546ce6517ca7e2f6] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference:
{[req][re_date]}"}
Are you using {[req][re_date]}
or [req][re_date]
? Should be the second one.
Here is a full example that works.
input {
generator {
lines => ['{"req": { "service": "ser1","re_date": "2020-09-10T09:14:38.479"}}']
count => 1
codec => json
}
}
filter {
json { source => "message"}
mutate { gsub => [ "[req][re_date]", "-", "/" ] }
}
output { stdout { codec => json_lines } }
I am using [req][re_date]. it seems mutate filter dose not recognize req.re_date as a field which is a key of json filter output. I used a simple in put as following:
{"request": "2020-09-10"}
and following logstash config:
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
mutate {
gsub => [
"{[request]}", "-", "/"
]
}
}
output {
elasticsearch {
hosts => ["0.0.0.0:9200"]
index => "test"
}
stdout { codec => rubydebug }
}
and still, logsatsh out put is as following:
also. I used following config and still output is not expected:
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
mutate {
gsub => [
"{%{request}}", "-", "/"
]
}
}
output {
elasticsearch {
hosts => ["0.0.0.0:9200"]
index => "test"
}
stdout { codec => rubydebug }
}
You keep putting { }
around the field name and that is what is messing it up.
Just use exactly like how I am and it works.
mutate { gsub => [ "[req][re_date]", "-", "/" ] }
system
(system)
Closed
May 26, 2021, 10:52am
9
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.