Jdbc input, no filters, csv output => CompatibilityError: incompatible encodings: UTF-8 and ASCII-8BIT

(Reuschling) #1

I read out sql server table data with jdbc input plugin, generating a csv file with csv output plugin. At some time (reproducable after a view minutes) it seems that an event has the content that yields to a CompatibilityError: incompatible encodings: UTF-8 and ASCII-8BIT.
The error takes place both with logstash 2.4.1 and 5.0.1 (with the csv fix from https://github.com/logstash-plugins/logstash-output-csv/issues/10).
I tried to set the codec=>plain to utf-8, no change. For me it seems that the csv module tries to create an ASCII file? I can not find any place where to adjust the output encoding. I work on an Linux system, the shell encoding inclusive LANG is set to UTF-8 also.

Encoding::CompatibilityError: incompatible encodings: UTF-8 and ASCII-8BIT>, :
backtrace=>["org/jruby/RubyArray.java:1892:in join'", "/logstash-5.0.1/vendor/jruby/lib/ruby/1.9/csv.rb:1729:in<<'",
"/logstash-5.0.1/vendor/jruby/lib/ruby/1.9/csv.rb:1269:in generate_line'", "/logstash-5.0.1/vendor/jruby/lib/ruby/1.9/csv.rb:2347:into_csv'",
"/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-csv-3.0.2/lib/logstash/outputs/csv.rb:45:in multi_receive_encoded'", "org/jruby/RubyArray.java:1613:ineach'",
"/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-csv-3.0.2/lib/logstash/outputs/csv.rb:37:in multi_receive_encoded'", "/logstash/logstash-core/lib/logstash/outputs/base.rb:90:inmulti_receive'",
"/logstash/logstash-core/lib/logstash/output_delegator_strategies/legacy.rb:19:in multi_receive'", "/logstash/logstash-core/lib/logstash/output_delegator.rb:42:inmulti_receive'",
"/logstash/logstash-core/lib/logstash/pipeline.rb:297:in output_batch'", "org/jruby/RubyHash.java:1342:ineach'",
"/logstash/logstash-core/lib/logstash/pipeline.rb:296:in output_batch'", "/logstash/logstash-core/lib/logstash/pipeline.rb:252:inworker_loop'",
"/logstash/logstash-core/lib/logstash/pipeline.rb:225:in `start_workers'"]}

(Reuschling) #2

the line of code in ruby csv.rb (not logstash csv.rb), first line of the stacktrace is

output = row.map(&@quote).join(@col_sep) + @row_sep # quote and separate

..so it seems that not the output encoding is the problem, but maybe the rows/according array entries are in different encodings? At least this would be possible in ruby, as I read (I'm not the ruby expert).

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.