I have a filebeat config to pick up a CSV file shown below:
paths: - /path/to/CSV multiline.pattern: '^\d' multiline.negate: true multiline.match: after
Here is a sample from the CSV:
RTime,Concept,Time,YestDate,YestCount,PrevDate,PrevCount,TodDate,TodCount 2019-08-12 14:10:39.993000000,WS,20:30,1900-01-01,0,2018-08-13,3,1900-01-01,0 2019-08-12 14:10:39.993000000,WS,21:00,1900-01-01,0,2018-08-13,2,1900-01-01,0 2019-08-12 14:10:39.993000000,WS,21:30,2019-08-11,1,2018-08-13,1,1900-01-01,0 2019-08-12 14:10:39.993000000,WS,Total,1900-01-01,717,1900-01-01,642,1900-01-01,375
The problem is that filebeat picks up the last couple of characters in the message field.
For ex, the above CSV sample picked up
01,375 instead of the full row.
I believe it has something to do with the delimeter but haven't been able to figure out exactly what's needed...
Could it because of the way the CSV is written to? I noticed that when I renamed the CSV and told filebeat to pick it up, it would pick up all the rows find but when something new got written, it would only pick up the last bit like shown above... maybe the csv has to write a new line?
Any help would be appreciated!