Incompatible encodings: IBM437 and UTF-8

Elastic Stack Logstash
1

Hi,

I just installed logstash 8.4.3 on Windows Server 2019 and copied over configs from 8.2.3. logstash doesn't start:

[2022-10-18T18:24:26,830][ERROR][logstash.outputs.elasticsearch][main][42989fb99e78bec7495dbe5bfe5a541ef44a4d6032e0d3cd83414ebb23817b03] An unknown error occurred sending a bulk request to Elasticsearch (will retry indefinitely) {:message=>"incompatible encodings: IBM437 and UTF-8", :exception=>Encoding::CompatibilityError, :backtrace=>["org/jruby/ext/stringio/StringIO.java:1162:in write'", "C:/Program Files/ELK/logstash/8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:142:in block in bulk'", "org/jruby/RubyArray.java:1865:in each'", "org/jruby/RubyEnumerable.java:1143:in each_with_index'", "C:/Program Files/ELK/logstash/8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:125:in bulk'", "C:/Program Files/ELK/logstash/8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:296:in safe_bulk'", "C:/Program Files/ELK/logstash/8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:228:in submit'", "C:/Program Files/ELK/logstash/8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:177:in retrying_submit'", "C:/Program Files/ELK/logstash/8.4.3/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.6.0-java/lib/logstash/outputs/elasticsearch.rb:342:in multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121:in multi_receive'", "C:/Program Files/ELK/logstash/8.4.3/logstash-core/lib/logstash/java_pipeline.rb:300:in block in start_workers'"]}`

8.2.3 still works fine.

Did I forget anything?

  • pipeline.id: main
    path.config: "C:/Program Files/ELK/logstash/8.4.3/config/conf.d/*.conf"

input {
beats {
host => localhost
port => 5044
}
}

output {
if [@metadata][pipeline] {
elasticsearch {
hosts => ["https://localhost:9200"]
ssl => true
ssl_certificate_verification => false
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
pipeline => "%{[@metadata][pipeline]}"
user =>
password =>
}
} else {
elasticsearch {
hosts => ["https://localhost:9200"]
ssl => true
ssl_certificate_verification => false
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user =>
password =>
}
}
}

Thank you!

1 Like
2

Solution at: https://stackoverflow.com/a/61399047/10309142

3 
input {
	beats {
		host => localhost
		port => 5044
		codec => plain { charset => "UTF-8"}
	}
	stdin {
		codec => plain { charset => "UTF-8"}
	}
}

output {
	if [@metadata][pipeline] {
		elasticsearch {
			hosts => ["https://localhost:9200"]
			ssl => true
			ssl_certificate_verification => false
			manage_template => false
			index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
			pipeline => "%{[@metadata][pipeline]}"
			user => <USER>
			password => <PASSWORD>
			codec => plain { charset => "UTF-8"}
		}
	} else {
		elasticsearch {
			hosts => ["https://localhost:9200"]
			ssl => true
			ssl_certificate_verification => false
			manage_template => false
			index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
			user => <USER>
			password => <PASSWORD>
			codec => plain { charset => "UTF-8"}
		}
	}
}

Still same error.

4

You can ask for help https://elasticstack.slack.com/archives/CNKF2D325, someone can help you!

5

LS and FB are using UTF-8 by default. Try with different charset, but since this data comes from FB, try to set IBM437 in filebeat.yml
codec => plain { charset => "IBM437"}

6

I'm using WInLogbeat and I don't see where to configure that.

Anyway, it workes fine with 8.2.3 and seems broken with 8.4.3 and I would like to know why. :slight_smile:

7

Have you try to set on LS side?

From filebeat.reference.yml. The winlogbeat.reference.yml doesn't have it.

 # Configure the file encoding for reading files with international characters
  # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
  # Some sample encodings:
  #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
  #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
  #encoding: plain
8

I seem to have the same issue, but only on all my windows desktops not on the windows server.
And it somehow magically goes away if I redirect standard output of logstash, which just makes no sense at all.

https://discuss.elastic.co/t/encoding-issue-depending-on-how-logstash-is-started/317136

9

Hi all, just to confirm that this issue still exists with Logstash 8.5.0.

10

Have you submitted the issue on github?

11

@Rios. A related issue was already submitted to Github by @simon137. See link below:

1 Like
12

Yes I have. See Steve's answer, thank you. Please feel free to give the issue a thumbs up or a comment :slight_smile:

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.