How can rename logstash fields

rename fields:


May be you need to clean first your message using gsub filter

Thanks for your help. I have done according your advise. All working better.

This is a small part of full message.

Can I convert this before clean message?

txnDateTime: 2020-11-04 17:7:22

Thanks again.

Something like this may help

mutate { add_field => { "event_date_time" => "%{[date][year]}-%{[date][month]}-%{[date][day]} ........................" }}

Thanks, I wish just replace the following message via gsub.



I can match
txnDateTime":{"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}} BY txnDateTime":([^]+)\w}}

But I can not match only numerical value from above message to get the following result.


Thanks again for helping me

There are differents possibilities with logstash, as i can se that this like a json field but not full json compliant, missed {" in the beginning and } at the end
So basicall you set a grok to math the full json message, I will support that you have a filed called message with the content of the incomplete json

message => txnDateTime":{"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}}

Something like this may help

#> txnDateTime":{"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}}
#> txnDateTime => {"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}}
#> {"txnDateTime":2020-11-4:17:7:22:0"}

input { stdin { } }

filter {
grok { match => { "message" => "txnDateTime\":%{GREEDYDATA:txnDateTime}"}}
json { source => "txnDateTime" }
mutate { add_field => { "DateTime" => "%{[date][year]}-%{[date][month]}-%{[date][day]} %{[time][hour]}:%{[time][minute]}:%{[time][second]}.%{[time][nano]}"}}
date {
    match => [ "DateTime" , "yyyy-M-d H:m:s.SSS" ]
	target => "@timestamp"
	timezone => "UTC"

output {
  stdout { codec => rubydebug }

Sorry, I didn't explain my problem.

My full message:
ConsumerRecord(topic = CB, partition = 16, leaderEpoch = 27, offset = 1515, CreateTime = 1605778874095, serialized key size = -1, serialized value size = 1137, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {,test:false,clientCbsSId:20201119_031112_S421672_11,txnDateTime:{date:{year:2020,month:11,day:19},time:{hour:15,minute:37,second:35,nano:0}},senderId:COMMUNITY,acctId:0010151403201,sendToNumber:1718207974,message:Community. Thank you.,clientRequestDateTime:{date:{year:2020,month:11,day:19},time:{hour:15,minute:41,second:14,nano:95000000}},expiryDateTime:{date:{year:2020,month:11,day:20},time:{hour:3,minute:42,second:12,nano:0}},retryCount:1,valid:true,validateCell:false,defaultSmsLength:160,exceedDefaultSmsLength:false,status:GW_PENDING,isDirty:false,customerNo:0151403,smsFormat:SMS_FORMAT_DEFAULT,unicode:0,includeResend:false,messagetype:0,smsResultList:[],version:0,active:1,tzName:UTC/GMT})

I wish to split all fields.
Thanks for helping me

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.