Hello,
I have logstash [6.7] complaining when I set "\r\n" as delimiter for the file input plugin.
The file I'd like logstash to handle is
file incident_sample.csv
incident_sample.csv: ASCII text, with very long lines, with CRLF, LF line terminators
It's a csv file, with some fields which are BLOBs dumps, containing LF characters. The record is actually terminated with CRLF (0x0d0x0a).
Logstash informs that cannot reach EOL, although there are more than 20 \r\n in sample file:
[2021-06-22T12:03:49,625][INFO ][filewatch.readmode.handlers.readfile] buffer_extract: a delimiter can't be found in current chunk, maybe there are no more delimiters or the delimiter is incorrect or the text before the delimiter, a 'line', is very large, if this message is logged often try increasing the `file_chunk_size` setting. {"delimiter"=>"\\r\\n", "read_position"=>32768, "bytes_read_count"=>28720, "last_known_file_size"=>61488, "file_path"=>"/tmp/incident_sample.csv"}
Please note that, if I replace \r\n with ###### in data file and use
input {
file {
path => [ "/tmp/incident_sample.csv" ]
sincedb_path => "/dev/null"
mode => "read"
delimiter => "######"
}
}
the lines are correctly recognized (and multiline fields retain their \n)
Besides replacing the record terminator in input files, is there any other solution for my problem?
TIA,
Paolo
PS
A similar unexpected handling of '\r\n' is met if I set that delimiter for the codec property:
input {
file {
codec => line {
charset => "ISO-8859-1"
delimiter => "\r\n" # does not work
} . . .