Base64 decode issue

[Rajesh_R]

Hi Team, Does Anyone know how to decode base64 in logstash?
Logfile :
{"ID":"11166946081959","Type":"LOG","pID":"rajesh.r@gmail.com","Interface":"Offboarding","payload": "PD94bWzCoHZlcnNpb249IjEuMCLCoGVuY29kaW5nPSJJU08tODg1OS0xIj8+wqDCoDxub3RlPsKgwqDCoDx0bz5Ub3ZlPC90bz7CoMKgwqDCoDxmcm9tPkphbmk8L2Zyb20+wqDCoMKgwqA8aGVhZGluZz5SZW1pbmRlcjwvaGVhZGluZz7CoMKgwqDCoDxib2R5PkRvbid0wqBmb3JnZXTCoG1lwqB0aGlzwqB3ZWVrZW5kITwvYm9keT7CoMKgPC9ub3RlPsKg"}

Config file:
input {
file{
path => "/home/rajesh/ELK/json.log"
start_position => beginning
tags => [ "time" ]
sincedb_path => "/dev/null"
codec => json {
target => "[document]"
}
}
}
filter {
ruby { code => 'event.set("decoded", Base64.decode64(event.get("payload")))' }
}
output {
if "time" in [tags] {
elasticsearch {
hosts => ["https://localhost:9200"]
cacert => '/home/rajesh/ELK/logstash-8.4.3/config/certs/http_ca.crt'
index => "time-%{+YYYY.MM.dd}"
user => "rajesh"
password => "rajesh"
}
}
}

[Badger]

@Rajesh_R

ruby { code => 'event.set("decoded", Base64.decode64(event.get("payload")))' }

That will decode the base64 and result in

"decoded" => "<?xml\xC2\xA0version=\"1.0\"\xC2\xA0encoding=\"ISO-8859-1\"?>\xC2\xA0\xC2\xA0<note>

The \xC2\xA0 are non-breaking spaces in UTF-8, but the decoded string is ASCII-8BIT (puts Base64.decode64(event.get("payload")).encoding.name will tell you that).

So the default encoding is wrong. You can change it using

ruby { code => 'event.set("decoded", Base64.decode64(event.get("payload")).force_encoding("UTF-8"))' }

which will result in

"decoded" => "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>  <note>   <to>Tove</to>...

Note that an xml filter will not parse non-breaking spaces, so you need to replace them if you want to parse the XML

mutate { gsub => [ "decoded", "[[:space:]]", " " ] }

[Rajesh_R]

I tried the same thing, but the decoded value not coming in kibana. please find the below snap for reference