# How to read base64 encoded logs from kafka through logstash pipeline
**URL:** https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688
**Category:** Logstash
**Created:** [December 20, 2023, 3:49am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688 "2023-12-20T03:49:10Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@upreddy](https://discuss.elastic.co/u/upreddy)
#### Post date: [December 20, 2023, 3:49am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/1 "2023-12-20T03:49:10Z")
Hello all,
I am working on something I have never worked on before and I really do not know where to go from here and I am hoping someone might have some direction for me to attempt trying to get this parsed to Elasticsearch properly.
The application team sending logs to kafka topics with base64 encoded. Now i want to decode that base64 encoded logs through logstash pipeline. Could you please guide me on this?
Thanks
Upendar
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 20, 2023, 4:16am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/2 "2023-12-20T04:16:35Z")
> [@upreddy](#):
>
> Now i want to decode that base64 encoded logs through logstash pipeline.
You will probably need a ruby filter. See [these](https://discuss.elastic.co/t/base64-decode-issue/319874/2) [two](https://discuss.elastic.co/t/can-logstash-base64-decode-then-parse-json-from-a-cloudfront-log/280367/2) threads.
---
### Author:  [@sholzhauer](https://discuss.elastic.co/u/sholzhauer)
#### Post date: [December 20, 2023, 9:07am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/3 "2023-12-20T09:07:27Z")
I think you are looking for the cipher filter, it has a base64 option.
> **[Cipher filter plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-cipher.html#plugins-filters-cipher-base64)**
---
### Author:  [@upreddy](https://discuss.elastic.co/u/upreddy)
#### Post date: [December 21, 2023, 1:03am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/4 "2023-12-21T01:03:53Z")
Hi Badger,
Thanks for replying!
I have tried below ruby filter
ruby { code =\> 'event.set("decoded", Base64.decode64(event.get("payload")).force\_encoding("UTF-8"))' }
But i am getting below error.
[ERROR] 2023-12-21 00:30:03.194 [[pecspayload]\>worker1] ruby - Ruby exception occurred: undefined method `unpack1' for nil:NilClass {:class=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/base64.rb:59:in `decode64'", "(ruby filter code):2:in `block in filter_method'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in `inline\_script'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `do\_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178:in `block inmulti_filter'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133:in `multi\_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:304:in `block in start\_workers'"]}
i am sharing my logstash configuration.
input
{
kafka
{
bootstrap\_servers =\> "A,B,C"
topics =\> "XX\_XX\_XX"
group\_id =\> "XX\_XX\_XX"
codec =\> json {}
security\_protocol =\> "SASL\_SSL"
sasl\_mechanism =\> "SCRAM-SHA-512"
sasl\_jaas\_config =\> "org.apache.kafka.common.security.scram.ScramLoginModule required username='XX'
password='XXXXXX';"
ssl\_endpoint\_identification\_algorithm =\> ""
}
}
filter
{
ruby { code =\> 'event.set("decoded", Base64.decode64(event.get("payload")).force\_encoding("UTF-8"))' }
}
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 21, 2023, 1:28am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/5 "2023-12-21T01:28:47Z")
> [@upreddy](#):
>
> undefined method `unpack1' for nil:NilClass {:class=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/base64.rb:59:in `decode64'
I think that means event.get("payload") returned nil, i.e, the field does not exists. Perhaps should use event.get("message"), or whatever field is Base64 encoded.
---
### Author:  [@upreddy](https://discuss.elastic.co/u/upreddy)
#### Post date: [December 21, 2023, 5:02am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/6 "2023-12-21T05:02:30Z")
Hi Badger,
I have used below ruby filter.
ruby { code =\> 'event.set("decoded", Base64.decode64(event.get("message")))' }
i am getting below result.
{"tags":["\_jsonparsefailure"],"message":"\u001F�\b\u0000\u0000\u0000\
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 21, 2023, 9:17am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/7 "2023-12-21T09:17:39Z")
If you ever get a � then the encoding is wrong. If you know what the encoding should be then you can use .force\_encoding to fix it.
If you do not know know what the encoding should be then it is unlikely that anyone will be able to answer if you ask about it.
---
### Author:  [@upreddy](https://discuss.elastic.co/u/upreddy)
#### Post date: [December 21, 2023, 9:28am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/8 "2023-12-21T09:28:31Z")
Hi Badger,
Thanks for answering!
I have checked with application team they are doing GZIP and encoding with uTF-8.
Now how can i read logs through logstash.
---
### Author:  [@upreddy](https://discuss.elastic.co/u/upreddy)
#### Post date: [January 3, 2024, 8:54am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/9 "2024-01-03T08:54:15Z")
Hi Team,
Any one please guide me on this?
Thanks
---
### Author:  [@grumo35](https://discuss.elastic.co/u/grumo35)
#### Post date: [January 3, 2024, 9:28am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/10 "2024-01-03T09:28:30Z")
Hello upreddy,
The ruby code snippet works it's just your data that is not encoded in full ascii readble.
Have you tried to decode it any other way is it readable ?
Base64 encoding is made to ship data with ASCII charcaters, if you try to decode it then render the binary back to ascii you might have some surprises.....
I'm not sure what's your problem is here ?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 31, 2024, 9:28am UTC](https://discuss.elastic.co/t/how-to-read-base64-encoded-logs-from-kafka-through-logstash-pipeline/349688/11 "2024-01-31T09:28:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.