Filebeat not picking up CSV properly

I have a filebeat config to pick up a CSV file shown below:

    - /path/to/CSV
  multiline.pattern: '^\d'
  multiline.negate: true
  multiline.match: after

Here is a sample from the CSV:

2019-08-12 14:10:39.993000000,WS,20:30,1900-01-01,0,2018-08-13,3,1900-01-01,0
2019-08-12 14:10:39.993000000,WS,21:00,1900-01-01,0,2018-08-13,2,1900-01-01,0
2019-08-12 14:10:39.993000000,WS,21:30,2019-08-11,1,2018-08-13,1,1900-01-01,0
2019-08-12 14:10:39.993000000,WS,Total,1900-01-01,717,1900-01-01,642,1900-01-01,375

The problem is that filebeat picks up the last couple of characters in the message field.
For ex, the above CSV sample picked up 01,375 instead of the full row.

I believe it has something to do with the delimeter but haven't been able to figure out exactly what's needed...

Could it because of the way the CSV is written to? I noticed that when I renamed the CSV and told filebeat to pick it up, it would pick up all the rows find but when something new got written, it would only pick up the last bit like shown above... maybe the csv has to write a new line?

Any help would be appreciated!


Could it because of the way the CSV is written to?

Potentially, yes. How exactly is the CSV written?

maybe the csv has to write a new line?


Filebeat tails a file, it does not send the complete log file. When tailing it first splits the log into multiple lines (based on \n or \r\n by default), and then applies the multiline filter.

After sending it remembers the last file offset and starts from this offset when the file size has increased.

The way you describe it, it sounds like there was already a newline written, but then "old file contents" gets overwritten. This would indeed break collection, as filebeat does not track overall file changes, but only the file size and the last read offset.

Thanks for the reply Steffen! So I had the developer who was generating the CSV add a newline character after every row so now its looking for^\n which i believe should work. However, we have ran into another unrelated issue where filebeat doesn't recognize the output section of the YML so it doesn't output the way I want it to... When I test config, it says Config OK but the output part of the YML is being completely ignored for some reason.