Logstash error when dictionary(CSV) has \r or \n at end of line

[Toni_Riera]

Hello there,

I'm developing a .conf related to security analysis with ELK Stack, and saw the article that Blueliv made on January '18. What i'm trying to do is load a csv file wich will serve as a dictionary to match malware with our logs.

The problem is that when loading the csv file, logstash gives me this error:

[ERROR] 2018-05-14 11:18:32.677 [[main]-pipeline-manager] pipeline - Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::FilterDelegator:0x2ddb204e @metric_events_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric_events_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric_events_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration_in_millis value:0, @id="ba05429333b650e6301b88bdb9162dfedd3189b4b081f87a314ca272aae3179f", @klass=LogStash::Filters::Translate, @metric_events=#<LogStash::Instrument::NamespacedMetric:0xf44cee3 @metric=#<LogStash::Instrument::Metric:0x2139f8e5 @collector=#<LogStash::Instrument::Collector:0x12e1173d @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x664c8c7f @store=#<Concurrent:0x00000000000fb0 entries=4 default_proc=nil>, @structured_lookup_mutex=#Mutex:0x4f567583, @fast_lookup=#<Concurrent:0x00000000000fb4 entries=95 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :ba05429333b650e6301b88bdb9162dfedd3189b4b081f87a314ca272aae3179f, :events]>, @filter=<LogStash::Filters::Translate field=>"url", destination=>"malware", dictionary_path=>"/etc/logstash/malware-utf-8.csv", id=>"ba05429333b650e6301b88bdb9162dfedd3189b4b081f87a314ca272aae3179f", enable_metric=>true, periodic_flush=>false, override=>false, refresh_interval=>300, exact=>true, regex=>false, refresh_behaviour=>"merge">>", :error=>"LogStash::Filters::Translate: Unquoted fields do not allow \r or \n (line 2294). when loading dictionary file at /etc/logstash/malware-utf-8.csv", :thread=>"#<Thread:0x3b04f1a6@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 run>"}

Basically, this "Unquoted fields do not allow \r or \n (line 2294)". The main thing, is that the .csv file at line 2294 which is end of line, does not have any \n or \r (no line empty after last line).

The end of line is this one:

2017/10/26_13:48,"photoscape.ch/Setup.exe","31.148.219.11","knigazdorovya.com.","trojan","-","14576","0","CZ",

We end with a comma, as we're constantly adding new malware lines. Could be that?

I've been checking some other topics in discuss, also in github and either i can't find anyting very similar to mine, or solutions that worked for other people.

Thank you very much, hope he can solve it.

[Toni_Riera]

Sorry this is my .conf part when loading the file:

input {
stdin {
codec => json
}
}

filter {
translate {
field => "url"
destination => "malware"
dictionary_path => "/etc/logstash/malware-utf-8.csv"
}
}

[magnusbaeck (Magnus Bäck)]

We end with a comma, as we're constantly adding new malware lines.

This doesn't make sense. In CSV, the comma separates columns. Lines are separated by newlines. Ending all lines (even the last one) with a newline is a good idea.

[Toni_Riera]

Yes, it does sepparate columns, i was meaning that at each end of line of csv there is a comma, so by not having an empty line at the end of file, the last character was a comma. Therefore, i solve it.

Saw an empty line at middle of document, and later the last error was due to a version of UTF.

Thanks!