Editing date time filed in json filter output

Hi all
I am using filebeat 7.8 which sends log to logstash 7.8. Each line of log is as following:

{"req": { "service": ser1,"re_date": "2020-09-10T09:14:38.479"}}
and my logstash filter is as following:

         input {
           beats {
           port => 5044
         }
         }
        filter {
            json {
            source => "message"
          }
        }
    	output {

      elasticsearch {
        hosts => ["0.0.0.0:9200"]
            index => "test"
            }
           stdout { codec => rubydebug }
          }

which will be index in elasticsearch as following event:

req.service: ser1
req.re_date: 2020-09-10T09:14:38.479

Now, i want to change req.re_date format to (2020/09/10T09:14:38.479) using following filter, but the output is still same and dose not make any change.

input {
  beats {
    port => 5044
  }
}
filter {
      json {
        source => "message"
      }
	  		mutate {
	gsub => [
		"%{req.re_date}", "-", "/"
		]
}
	
    }
	output {

  elasticsearch {
    hosts => ["0.0.0.0:9200"]
        index => "test"
        }

stdout { codec => rubydebug }
  }

Any advise will be so appreciated.

Regards

Try gsub => [ "[req][re_date]", "-", "/" ]

1 Like

Many thanks for your reply. But, output is still the same as "2020-09-10T09:14:38.479". Following warning has been found in the log:

[2021-04-24T17:34:28,262][WARN ][logstash.filters.mutate ][main][e1ebc65d12b5ed0ee94689d88d19899b9e56d1373846c87a546ce6517ca7e2f6] Exception caught while applying mutate filter {:exception=>"Invalid FieldReference: {[req][re_date]}"}

Are you using {[req][re_date]} or [req][re_date]? Should be the second one.

Here is a full example that works.

input {
   generator {
     lines => ['{"req": { "service": "ser1","re_date": "2020-09-10T09:14:38.479"}}']
     count => 1
     codec => json
   } 
}
filter {
    json { source => "message"}
    mutate { gsub => [ "[req][re_date]", "-", "/" ] }
}
output { stdout { codec =>  json_lines } }

I am using [req][re_date]. it seems mutate filter dose not recognize req.re_date as a field which is a key of json filter output. I used a simple in put as following:

{"request": "2020-09-10"}

and following logstash config:

input {
  beats {
    port => 5044
  }
}
filter {
      json {
        source => "message"
      }
	  		mutate {
	gsub => [
		"{[request]}", "-", "/"
		]
}



    }
	output {

  elasticsearch {
    hosts => ["0.0.0.0:9200"]
        index => "test"
        }

stdout { codec => rubydebug }
  }

and still, logsatsh out put is as following:
image

also. I used following config and still output is not expected:

input {
  beats {
    port => 5044
  }
}
filter {
      json {
        source => "message"
      }
	  		mutate {
	gsub => [
		"{%{request}}", "-", "/"
		]
}
    }
	output {
  elasticsearch {
    hosts => ["0.0.0.0:9200"]
        index => "test"
        }
stdout { codec => rubydebug }
  }

You keep putting { } around the field name and that is what is messing it up.

Just use exactly like how I am and it works.

mutate { gsub => [ "[req][re_date]", "-", "/" ] }

Many thanks. It works

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.