8.4.2: "incompatible encodings: CP850 and UTF-8" - Issue sending DB records to Elastic after upgrade to Elastic 8.4.2 from 8.3.2

Elastic Stack Logstash
1

Hi All,

I've been running the 8.3.2 Elastic products for some months (ELK), using logstash to poll an MSSQL DB regularly and pull the data into Elasticsearch.

This has been working without any significant problems.

I recently upgraded one of our environments to 8.4.2 and am seeing the following error intermittently in the Logstash console/log file.

The error continues to be repeated by Logstash and no further records are pulled from the DB to Elasticsearch. I haven't been able to identify any single particular record within the DB that would seem to be 'odd' and might cause this so I'm not sure if it's an input data issue or if it's an issue between Logstash and Elastic.
The same DB data is processed successfully by the 8.3.2 product versions.

Below is the error I see from Logstash:

[2022-10-03T11:06:02,468][ERROR][logstash.outputs.elasticsearch][main][7930e5731f7d543c5f1986a323e12f921fab2fadd6e56bc484ccee7af22a8242] An unknown error occurred sending a bulk request to Elasticsearch (will retry indefinitely) {:message=>"incompatible encodings: CP850 and UTF-8", :exception=>Encoding::CompatibilityError, :backtrace=>["org/jruby/ext/stringio/StringIO.java:1162:in write'", "C:/APPS/ELASTIC/logstash-8.4.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:142:in block in bulk'", "org/jruby/RubyArray.java:1865:in each'", "org/jruby/RubyEnumerable.java:1143:in each_with_index'", "C:/APPS/ELASTIC/logstash-8.4.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:125:in bulk'", "C:/APPS/ELASTIC/logstash-8.4.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:296:in safe_bulk'", "C:/APPS/ELASTIC/logstash-8.4.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:228:in submit'", "C:/APPS/ELASTIC/logstash-8.4.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:177:in retrying_submit'", "C:/APPS/ELASTIC/logstash-8.4.2/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:342:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "C:/APPS/ELASTIC/logstash-8.4.2/logstash-core/lib/logstash/java_pipeline.rb:300:in block in start_workers'"]}`

If anyone has any insight as to what the problem might be that would be really helpful.

Thanks,
Steve

1 Like
2

I seem to have the same issue, but only on all my windows desktops not on the windows server.
And it somehow magically goes away if I redirect standard output of logstash, which just makes no sense at all.

https://discuss.elastic.co/t/encoding-issue-depending-on-how-logstash-is-started/317136

3

I created an issue on github: Logstash throws "Incompatible Encodings" error when querying NVARCHAR-Fields from MSSQL-Server · Issue #14679 · elastic/logstash · GitHub

4

thanks @simon137 for your comments. I've had a look at your own post, and at the GIT change that you pointed at as a likely suspect - it is definitely around the character encoding area so I think you may be correct.

Hopefully this will get some attention and patched, for the time being I'll revert back to 8.3.x and stick with what was working for us.

Regards,
Steve

1 Like
5

Just tested Logstash 8.5.0 and the issue is present there too.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.