codec => plain { charset => "UTF-8"}
}
}
}
```
Still same error.
---
### Author:  [@dong.nguyen.hdkt](https://discuss.elastic.co/u/dong.nguyen.hdkt)
#### Post date: [October 20, 2022, 1:29am UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/4 "2022-10-20T01:29:18Z")
You can ask for help [https://elasticstack.slack.com/archives/CNKF2D325](https://elasticstack.slack.com/archives/CNKF2D325), someone can help you!
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [October 20, 2022, 11:10am UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/5 "2022-10-20T11:10:27Z")
> [@Helmut](#):
>
> `codec => plain { charset => "UTF-8"}`
LS and FB are using UTF-8 by default. Try with different charset, but since this data comes from FB, try to set IBM437 in filebeat.yml
`codec => plain { charset => "IBM437"}`
---
### Author:  [@Helmut](https://discuss.elastic.co/u/Helmut)
#### Post date: [October 20, 2022, 12:17pm UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/6 "2022-10-20T12:17:55Z")
I'm using WInLogbeat and I don't see where to configure that.
Anyway, it workes fine with 8.2.3 and seems broken with 8.4.3 and I would like to know why. π
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [October 20, 2022, 12:29pm UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/7 "2022-10-20T12:29:36Z")
Have you try to set on LS side?
From filebeat.reference.yml. The [winlogbeat.reference.yml](https://github.com/elastic/beats/edit/main/libbeat/docs/reference-yml.asciidoc) doesn't have it.
```auto
# Configure the file encoding for reading files with international characters
# following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
# Some sample encodings:
# plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
# hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
#encoding: plain
```
---
### Author:  [@simon137](https://discuss.elastic.co/u/simon137)
#### Post date: [October 20, 2022, 6:08pm UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/8 "2022-10-20T18:08:52Z")
I seem to have the same issue, but only on all my windows desktops not on the windows server.
And it somehow magically goes away if I redirect standard output of logstash, which just makes no sense at all.
[https://discuss.elastic.co/t/encoding-issue-depending-on-how-logstash-is-started/317136](https://discuss.elastic.co/t/encoding-issue-depending-on-how-logstash-is-started/317136)
---
### Author:  [@stevedearl](https://discuss.elastic.co/u/stevedearl)
#### Post date: [November 8, 2022, 8:58am UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/9 "2022-11-08T08:58:38Z")
Hi all, just to confirm that this issue still exists with Logstash 8.5.0.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [November 9, 2022, 4:51pm UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/10 "2022-11-09T16:51:32Z")
Have you submitted the issue on github?
---
### Author:  [@stevedearl](https://discuss.elastic.co/u/stevedearl)
#### Post date: [November 10, 2022, 3:29pm UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/11 "2022-11-10T15:29:10Z")
@Rios. A related issue was already submitted to Github by @simon137. See link below:
>
>
> \*\*Logstash information\*\*:
> 1. Logstash version: 8.4.3 (i checked the issue came β¦in from v8.3.3 to v8.4.0)
> 2. Logstash installation source: expanded from tar or zip archive
> 3. How is Logstash being run: via commandline or as a windows service
>
> \*\*Plugins installed\*\*: no extra plugins were installed
>
> \*\*JVM\*\* (e.g. \`java -version\`): Bundled JDK:
> openjdk 17.0.4 2022-07-19
> OpenJDK Runtime Environment Temurin-17.0.4+8 (build 17.0.4+8)
> OpenJDK 64-Bit Server VM Temurin-17.0.4+8 (build 17.0.4+8, mixed mode, sharing)
> \-\> but also tested with:
> openjdk 11.0.15 2022-04-19
> OpenJDK Runtime Environment Temurin-11.0.15+10 (build 11.0.15+10)
> OpenJDK 64-Bit Server VM Temurin-11.0.15+10 (build 11.0.15+10, mixed mode)
>
> \*\*OS version\*\*: Windows 10
>
> \*\*Description of the problem including expected versus actual behavior\*\*:
>
> If I query some some NVARCHAR-Fields from a Microsoft-SQL-Server (which in MSSQL-Server are always encoded in UTF-16) via logstash-jdbc-input plugin without specifying any special encoding or charset settings, neither in the input nor in the output logstash plugins, logstash failes to transfer the events to ElasticSearch by throwing this error over and over again for every document:
> \`\`\`\`
> \[2022-10-20T17:01:57,061\]\[ERROR\]\[logstash.outputs.elasticsearch\]\[index\_name\]\[9648a8b8c103d11863b72d1b6d9624b2c3b8d672ae4baf73a17af87e6cc0c3e7\]
> An unknown error occurred sending a bulk request to Elasticsearch (will retry indefinitely) {:message=\>"incompatible encodings: CP850 and UTF-8", :exception=\>Encoding::CompatibilityError,
> :backtrace=\>\[
> "org/jruby/ext/stringio/StringIO.java:1162:in \`write'",
> "C:/Program Files/ElasticSearch/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http\_client.rb:142:in \`block in bulk'",
> "org/jruby/RubyArray.java:1865:in \`each'",
> "org/jruby/RubyEnumerable.java:1143:in \`each\_with\_index'",
> "C:/Program Files/ElasticSearch/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http\_client.rb:125:in \`bulk'",
> "C:/Program Files/ElasticSearch/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin\_mixins/elasticsearch/common.rb:296:in \`safe\_bulk'",
> "C:/Program Files/ElasticSearch/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin\_mixins/elasticsearch/common.rb:228:in \`submit'",
> "C:/Program Files/ElasticSearch/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin\_mixins/elasticsearch/common.rb:177:in \`retrying\_submit'",
> "C:/Program Files/ElasticSearch/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:342:in \`multi\_receive'",
> "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in \`multi\_receive'", "C:/Program Files/ElasticSearch/logstash/logstash-core/lib/logstash/java\_pipeline.rb:300:in \`block in start\_workers'"\]}
> \`\`\`\`
>
> This worked fine up until Version 8.3.3 of Logstash, since Version 8.4.0 it doesn't work anymore.
> I also tried specifying the Encoding as UTF-16 with jdbc-input-plugins columns\_charset option, but this doesn't affect the behaviour of logstash at all.
>
> \*\*Steps to reproduce\*\*:
>
> 1. Create a pipeline with the following input.conf (you have to change the Connection String to any Microsoft SQL-Server of course):
> \`\`\`\`
> input {
> jdbc {
> jdbc\_driver\_library =\> "C:\\\\ProgramData\\\\ElasticSearch\\\\logstash\\\\drivers\\\\mssql-jdbc-10.2.0.jre8.jar"
> jdbc\_driver\_class =\> "com.microsoft.sqlserver.jdbc.SQLServerDriver"
> jdbc\_connection\_string =\> "jdbc:sqlserver://server\\instance;databasename=database;trustServerCertificate=true"
> jdbc\_default\_timezone =\> "Europe/Berlin"
> jdbc\_user =\> "user"
> jdbc\_password =\> "pw"
> schedule =\> "\*/5 6-19 \* \* \*"
> statement\_filepath =\> "C:\\\\ProgramData\\\\ElasticSearch\\\\logstash\\\\pipelines\\\\index\_name\\\\queries\\\\sqlQuery.sql"
> clean\_run =\> false
> use\_column\_value =\> true
> tracking\_column =\> "editdate"
> tracking\_column\_type =\> "timestamp"
> last\_run\_metadata\_path =\> "C:\\\\ProgramData\\\\ElasticSearch\\\\logstash\\\\pipelines\\\\index\_name\\\\.logstash\_jdbc\_last\_run"
> }
> }
> \`\`\`\`
> 2. In the "sqlQuery.sql" just write any SQL-Query that queries at least one NVARCHAR-Field from any table.
> 3. Create the following output.conf for the pipeline (with different user/pw of course):
> \`\`\`\`
> output {
> elasticsearch {
> hosts =\> \["http://localhost:9200" \]
> index =\> "index\_name"
> document\_id =\> "%{document\_id}"
> action =\> "update"
> doc\_as\_upsert =\> true
> data\_stream =\> "false"
> user =\> "elastic"
> password =\> "pw"
> }
> }
> \`\`\`\`
> 4. Run logstash either manually from the commandline or as a windows service and you'll get the above error.
>
> (\*\*Oddly enough\*\* if you run the logstash.bat from the commandline and redirect stdout (and/or) stderr to a file, it works perfectly without any errors and indexes everything as it should. I have no idea how it is possible though, that output redirection affects the behaviour of logstash here, to be honest it just makes no sense.)
---
### Author:  [@simon137](https://discuss.elastic.co/u/simon137)
#### Post date: [November 10, 2022, 3:48pm UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/12 "2022-11-10T15:48:06Z")
Yes I have. See Steve's answer, thank you. Please feel free to give the issue a thumbs up or a comment π
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 8, 2022, 3:48pm UTC](https://discuss.elastic.co/t/incompatible-encodings-ibm437-and-utf-8/316918/13 "2022-12-08T15:48:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.