Translate v3.2.3 on LS 6.5.0: does not work anymore with .csv file

Elastic Stack Logstash
1

Dear all,

I'm trying to migrate a pipeline from LS 6.3.2 to 6.5.0 but there's a problem during its execution due to an error on the translate filter (used with a .csv file): "Unquoted fields do not allow \r or \n".

[2018-12-03T11:44:11,680][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/cpdlc_template
[2018-12-03T11:44:11,697][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"cpdlc-main", :plugin=>"#<LogStash::FilterDelegator:0x141b418 @metric_events_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric_events_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric_events_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration_in_millis value:0, @id="04f903c8cbc83b41265f9af525ef563877fcf096f2efba2a0153755bc6b83e4d", @klass=LogStash::Filters::Translate, @metric_events=#LogStash::Instrument::NamespacedMetric:0x1f371ec1, @filter=<LogStash::Filters::Translate destination=>"ADES_Country", dictionary_path=>"/data_bkp/logstash/00_Common/translate/countries.csv", exact=>true, regex=>true, id=>"04f903c8cbc83b41265f9af525ef563877fcf096f2efba2a0153755bc6b83e4d", field=>"ADES", enable_metric=>true, periodic_flush=>false, override=>false, refresh_interval=>300, refresh_behaviour=>"merge">>", :error=>"Translate: Unquoted fields do not allow \r or \n (line 1). when loading dictionary file at /data_bkp/logstash/00_Common/translate/countries.csv", :thread=>"#<Thread:0x40824a07@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:157 run>"}

  • Environment: Linux
  • LS 6.5.0 on Docker
  • translate file: countries.csv: example of content

^AG[A-Z]+$,Solomon Islands

^AN[A-Z]+$,Nauru

Any clue about that?
I didn't change anything on the .csv file.
Thanks in advance.

Michel

2

The same thing happened to us after upgrading from 6.4.2 to 6.5.3. The release notes for 6.5 allude to it: Translate filter

I had to to 2 things to fix it:

  1. enclose the items in double quotes
    change lookupvalue, translatevalue
    to
    "lookupvalue","translatevalue"
  2. run dos2unix on the csv file. The file was generated on a windows machine. We run logstash on linux with csv lookup tables. Don't know about windows lookup tables yet.

I don't know how it works with regex.

Larry

3

Many thanks, Larry for your message.

In fact, I succeeded in making it work: I just converted the .csv file build on Windows by editing it in Wordpad++ and selecting the option "Edit > EOL conversion > Unix (LF)" and saving it.

The problem was due to the end of line character which is not the same between Windows and Unix.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.