# Character encoding problems Filebeat & Logstash

**URL:** <https://discuss.elastic.co/t/character-encoding-problems-filebeat-logstash/150353>\
**Category:** Logstash\
**Created:** [September 28, 2018, 1:47pm UTC](https://discuss.elastic.co/t/character-encoding-problems-filebeat-logstash/150353 "2018-09-28T13:47:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![keysersozee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keysersozee/32/31563_2.png) [@keysersozee](https://discuss.elastic.co/u/keysersozee)\
**Post date:** [September 28, 2018, 1:47pm UTC](https://discuss.elastic.co/t/character-encoding-problems-filebeat-logstash/150353/1 "2018-09-28T13:47:00Z")

</div>

Hello,

I've read many topics in about this problem but it's still not resolved for me.

I have a fileabeat 6.2.2 installed on Windows, it will send logs logs to a Logstash 6.2.2 installed on Linux.

This my filebeat config:

```
- type: log
  scan_freqency: 10s  
  paths:
     - C:\logs\example.log
  encoding: utf-8

```

Logstash config:

```
input {
  beats {
    port => 5044
    codec => json { charset => "UTF-8" }
  }  
}

    output {
      http {
        automatic_retries => 10
        content_type => "application/json"
        format => "json"
        http_method => "post"
        ignorable_codes => 409
        keepalive => true
        url => "http://localhost:9090"
      }
      
      file {
        path => "D:\\result.txt"
    	codec => json { charset => "UTF-8" }
      }
    }

```

Logs are :

```
2018-09-26 16:05 - First: é, Second: è
2018-09-26 16:06 - Third: à, Fourth: â

```

Results in my **web application** (linked to a database) and in the file **D:\result.txt** :

```
2018-09-26 16:05 - First: ├®, Second: ├¿
2018-09-26 16:06 - Third: à, Fourth: â

```

Results from the Console ( **Ruby debug output** ) :

```
2018-09-26 16:05 - First: é, Second: è
2018-09-26 16:06 - Third: à, Fourth: â

```

I tried many encoding charsets following theses links :  
[https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json.html](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-json.html)  
[https://www.elastic.co/guide/en/logstash/current/plugins-codecs-plain.html](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-plain.html)

But I always get the same issue.

I also changed configuration :

- removed **encoding: utf-8** from filebeat configuration
- removed **codec =\> json{ charset =\> "UTF-8" }** from Logstash config

And nothing worked..

Can anyone help on this ? Thanks

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [September 28, 2018, 5:11pm UTC](https://discuss.elastic.co/t/character-encoding-problems-filebeat-logstash/150353/2 "2018-09-28T17:11:57Z")

</div>

Maybe the windows files are in a Microsoft encoding?

I can't say for filebeat, but charset setting in a codec is a **`from`** setting, meaning that, say you have a file in CP1252 encoding (Windows) and Logstash/Elasticsearch must have and expects UTF8 then you set the charset setting to "CP1252".

Here you are saying, "I know I have X encoding so please convert it to UTF8".

A few people have tried "universal string encoding detection" of an arbitrary piece of text but most have failed because the confidence level of the detection is a function of string length and the number of occurrences of multi-byte sequences.

So Logstash does not know what the source charset of the input data is. You can try ASCII\_8BIT because then LS will force encode to UTF8 and will replace any illegal UTF8 sequences with a � character.

---

<div class="post-metadata">

**Author:** ![keysersozee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keysersozee/32/31563_2.png) [@keysersozee](https://discuss.elastic.co/u/keysersozee)\
**Post date:** [October 11, 2018, 1:58pm UTC](https://discuss.elastic.co/t/character-encoding-problems-filebeat-logstash/150353/3 "2018-10-11T13:58:05Z")

</div>

Thank you for the reply,

Yes, the Windows file is in ANSI encoding.  
I had to set encoding to ANSI\_X3.4-1968:

```
filebeat.prospectors:
- type: log
  enabled: true
  encoding: ANSI_X3.4-1968
  paths:
    - C:\logs\example.log

```

PS: to people using other components to (enrich or store data), you need to also configure the message default converter.

For example in a **Spring** Web application, if we need to interact with Logstash, we usually use [RestTemplate](https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/client/RestTemplate.html)

All we need to do is :to set UTF-8 as the default charset :

```
restTemplate.getMessageConverters().add(0, new StringHttpMessageConverter(Charset.forName("UTF-8")));
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 8, 2018, 2:04pm UTC](https://discuss.elastic.co/t/character-encoding-problems-filebeat-logstash/150353/4 "2018-11-08T14:04:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.