Filebeat isn’t shipping the last line of a file

Hi everyone,

I'm new in filebeat, I'm using filebeat+ingest node pipeline to send csv properties into elasticsearch. Unfortunately filebeat doesn't send last line of my csv file. I read https://www.elastic.co/guide/en/beats/filebeat/current/newline-character-required-eof.html
but I haven't got idea how to handle it. Could you give me some tips how to resolve this case?
Is it possible to fix it using filebeat, ingest pipeline config or I have to add newline character manually?

My example config:

filebeat.yml

filebeat.inputs:
- type: log
  paths:
     - /data/*.csv
  pipeline: "my_pipe"
  index: "my_index"

data.csv

 xyz|xyz
 xyz|xyz
 xyz|xyz

my_pipe

PUT _ingest/pipeline/my_pipe
    {
    "processors": [
    {
        "grok": {
        "field": "message",
        "patterns": [
            "%{DATA:val1}\\|%{DATA:val2}"
        ]
        }
    },
   {
     "remove":{
        "field":"message"
     }
  }
    ],
    "on_failure" : [
        {
        "set" : {
            "field" : "error",
            "value" : "Error processing message"
        }
        }
    ]
    }

Thanks,
jack5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.