File input plugin - Delimiter property not working

I have logstash [6.7] complaining when I set "\r\n" as delimiter for the file input plugin.

The file I'd like logstash to handle is

file incident_sample.csv
incident_sample.csv: ASCII text, with very long lines, with CRLF, LF line terminators

It's a csv file, with some fields which are BLOBs dumps, containing LF characters. The record is actually terminated with CRLF (0x0d0x0a).

Logstash informs that cannot reach EOL, although there are more than 20 \r\n in sample file:

[2021-06-22T12:03:49,625][INFO ][filewatch.readmode.handlers.readfile] buffer_extract: a delimiter can't be found in current chunk, maybe there are no more delimiters or the delimiter is incorrect or the text before the delimiter, a 'line', is very large, if this message is logged often try increasing the `file_chunk_size` setting. {"delimiter"=>"\\r\\n", "read_position"=>32768, "bytes_read_count"=>28720, "last_known_file_size"=>61488, "file_path"=>"/tmp/incident_sample.csv"}

Please note that, if I replace \r\n with ###### in data file and use

input {
    file {
        path => [ "/tmp/incident_sample.csv" ]
        sincedb_path => "/dev/null"
        mode => "read"
        delimiter => "######"

the lines are correctly recognized (and multiline fields retain their \n)

Besides replacing the record terminator in input files, is there any other solution for my problem?


A similar unexpected handling of '\r\n' is met if I set that delimiter for the codec property:

input {
  file {
    codec => line {
        charset => "ISO-8859-1"
        delimiter => "\r\n"   # does not work
    } . . . 

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.