# Logstash+Kafka charset issue

**URL:** <https://discuss.elastic.co/t/logstash-kafka-charset-issue/29585>\
**Category:** Logstash\
**Created:** [September 18, 2015, 6:38pm UTC](https://discuss.elastic.co/t/logstash-kafka-charset-issue/29585 "2015-09-18T18:38:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ophelan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ophelan/32/4848_2.png) [@ophelan](https://discuss.elastic.co/u/ophelan)\
**Post date:** [September 18, 2015, 6:38pm UTC](https://discuss.elastic.co/t/logstash-kafka-charset-issue/29585/1 "2015-09-18T18:38:05Z")

</div>

I'm sending JSON formatted logs from logstash -\> HTTP endpoint -\> Kafka -\> logstash -\> ElasticSearch. The logs appear to be formatted correctly until ingested by the second logstash process. I've checked with another kafka consumer and the data looks fine there. Any ideas where things are getting mangled and how I can avoid?

Log rewritten by logstash client: {"@timestamp":"2015-09-18T16:41:56.706Z","@source\_host":"aws-mgmt-monitor-riemann-i-689515b8","@message":"Sep 18 16:41:56 dwhelan: test2","@fields":{"facility":"user","severity":"notice","program":"dwhelan","processid":"-","message":" test2"},"@version":"1","host":"[aws-mgmt-monitor-riemann-i-689515b8.xyzxyz-mgmt.com](http://aws-mgmt-monitor-riemann-i-689515b8.xyzxyz-mgmt.com)","path":"/var/log/json","type":"syslog"}  
Log written by logstash server: {"message":"😴\u0002\u0000\u0000\u0000\u0017\u0000\u0000\u00019\xF1\b:)\n\u0001\xFA\x87received%\u0001'x)~%\xB2\xFB\u0017\u0000\xF4W\x89@timestampW2015-09-18T16:35:52.678Z\x8B@source\_hostbaws-mgmt-monitor-riemann-i-689515b8\x87@message]Sep 18 O\u0000\xF30 dwhelan: test4\x86@fields\xFA\x87facilityCuser\x87severityEnotice\x86programF\>\u0000ӈprocessid@-\x86j\u0000\u0012ER\u0000\xD0\xFB\x87@version@1\x83\xAF\u0000\u001Fr\xAF\u0000\u0010q.xyzxyz\xD6\u0000\xF0\[u0015.com](http://u0015.com)\x83pathL/var/log/json\x83typeEsyslog\xFB\u00160\xE9\xEA","@version":"1","@timestamp":"2015-09-18T16:35:53.659Z"}

Client config:  
input {  
file {  
path =\> "/var/log/json\*"  
exclude =\> "\*.gz"  
type =\> "syslog"  
codec =\> "json"  
start\_position =\> "beginning"  
}  
}  
output {  
http {  
content\_type =\> "application/json; charset=utf-8"  
http\_method =\> "post"  
url =\> "[http://rt-metrics.xyxyz.com/events/logs](http://rt-metrics.xyxyz.com/events/logs)"  
headers =\> ["Authorization", "Basic AUTHSTRINGGOESHERE"]  
}  
}

Server config:  
input {  
kafka {  
consumer\_threads =\> 1  
topic\_id =\> "logs"  
zk\_connect =\> "[zk.xyzxyz.com:2181/logs](http://zk.xyzxyz.com:2181/logs)"  
codec =\> json {  
charset =\> "UTF-8"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:28am UTC](https://discuss.elastic.co/t/logstash-kafka-charset-issue/29585/2 "2017-07-06T05:28:42Z")

</div>