Hi everyone,
I'm new in filebeat, I'm using filebeat+ingest node pipeline to send csv properties into elasticsearch. Unfortunately filebeat doesn't send last line of my csv file. I read https://www.elastic.co/guide/en/beats/filebeat/current/newline-character-required-eof.html
but I haven't got idea how to handle it. Could you give me some tips how to resolve this case?
Is it possible to fix it using filebeat, ingest pipeline config or I have to add newline character manually?
My example config:
filebeat.yml
filebeat.inputs:
- type: log
paths:
- /data/*.csv
pipeline: "my_pipe"
index: "my_index"
data.csv
xyz|xyz
xyz|xyz
xyz|xyz
my_pipe
PUT _ingest/pipeline/my_pipe
{
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{DATA:val1}\\|%{DATA:val2}"
]
}
},
{
"remove":{
"field":"message"
}
}
],
"on_failure" : [
{
"set" : {
"field" : "error",
"value" : "Error processing message"
}
}
]
}
Thanks,
jack5