How can rename logstash fields

Elastic Stack Logstash
1

rename fields:

image
image804×501 10.4 KB

Thanks

2

May be you need to clean first your message using gsub filter

3

Thanks for your help. I have done according your advise. All working better.

This is a small part of full message.
txnDateTime":{"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}}

Can I convert this before clean message?
From:
txnDateTime":{"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}}

To
txnDateTime: 2020-11-04 17:7:22

Thanks again.

4

Something like this may help

mutate { add_field => { "event_date_time" => "%{[date][year]}-%{[date][month]}-%{[date][day]} ........................" }}
5

Thanks, I wish just replace the following message via gsub.

From:
txnDateTime":{"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}}

To
txnDateTime":2020-11-4:17:7:22:0}}

I can match
txnDateTime":{"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}} BY txnDateTime":([^]+)\w}}

But I can not match only numerical value from above message to get the following result.

txnDateTime":2020-11-4:17:7:22:0}}

Thanks again for helping me

6

There are differents possibilities with logstash, as i can se that this like a json field but not full json compliant, missed {" in the beginning and } at the end
So basicall you set a grok to math the full json message, I will support that you have a filed called message with the content of the incomplete json

message => txnDateTime":{"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}}

Something like this may help

#> txnDateTime":{"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}}
#> txnDateTime => {"date":{"year":2020,"month":11,"day":4},"time":{"hour":17,"minute":7,"second":22,"nano":0}}
#> {"txnDateTime":2020-11-4:17:7:22:0"}

input { stdin { } }

filter {
grok { match => { "message" => "txnDateTime\":%{GREEDYDATA:txnDateTime}"}}
json { source => "txnDateTime" }
mutate { add_field => { "DateTime" => "%{[date][year]}-%{[date][month]}-%{[date][day]} %{[time][hour]}:%{[time][minute]}:%{[time][second]}.%{[time][nano]}"}}
date {
    match => [ "DateTime" , "yyyy-M-d H:m:s.SSS" ]
	target => "@timestamp"
	timezone => "UTC"
  }
}

output {
  stdout { codec => rubydebug }
}
7

Sorry, I didn't explain my problem.

My full message:
ConsumerRecord(topic = CB, partition = 16, leaderEpoch = 27, offset = 1515, CreateTime = 1605778874095, serialized key size = -1, serialized value size = 1137, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = {UserName:usersms@gmail.com,test:false,clientCbsSId:20201119_031112_S421672_11,txnDateTime:{date:{year:2020,month:11,day:19},time:{hour:15,minute:37,second:35,nano:0}},senderId:COMMUNITY,acctId:0010151403201,sendToNumber:1718207974,message:Community. Thank you.,clientRequestDateTime:{date:{year:2020,month:11,day:19},time:{hour:15,minute:41,second:14,nano:95000000}},expiryDateTime:{date:{year:2020,month:11,day:20},time:{hour:3,minute:42,second:12,nano:0}},retryCount:1,valid:true,validateCell:false,defaultSmsLength:160,exceedDefaultSmsLength:false,status:GW_PENDING,isDirty:false,customerNo:0151403,smsFormat:SMS_FORMAT_DEFAULT,unicode:0,includeResend:false,messagetype:0,smsResultList:[],version:0,active:1,tzName:UTC/GMT})

I wish to split all fields.
Thanks for helping me

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.