Send the updated lines of file to logstash without re-running the filebeat

[npasha (Nadeem Pasha)]

I configured the filebeat.yml with the prospector and sends the output to logstash parse and then send to elastic search.
All worked fine.
My question is if i append or modify the file, is it possible to send only the added lines to logstash without rerunning the filebeat?

[Mojster]

Filebeat marks the file end. So if new line are added it takes only this ones.
Also if you stop/start filebeat it will continue.

I assume the added lines should be at the bottom, otherwise it won't work.

[npasha (Nadeem Pasha)]

thanks for the reply,

do i need to set "scan_frequency" in filebeat.yml to look for the new data?

[npasha (Nadeem Pasha)]

->I tried by adding a new line but did not updated i.e. did not sent the new line to logstash.

I added at the bottom of the file.

->When we stop the filebeat, how can it continue to send the data.

this is my filebeat.yml config file
filebeat.prospectors:

• type: log

  paths:

  • c:\Users\Nadeem.Pasha\Desktop\LOGS\proform-3.3.log.*
    scan_frequency: 10s

and logstashconfiguration file is

input {
beats {
port => 5044
}
}

filter {

}

output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
stdout { codec => rubydebug }
}

[Mojster]

Filebeat overview in general:
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html
More explained:
https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html

By logic, filebeat must be running to continue shipping data.

[Mojster]

No you don't need to set scan_frequency.
I'm always running on default.

[npasha (Nadeem Pasha)]

Thanks a lot for the reply,

filebeat is sending the data to logstash properly when i add some contents at the end,
but for some files, i can see twice the data is collected in logstash, may i know why this happens?

and why the error "failed to publish event...." as highlighted is coming?

[Mojster]

I have never had duplicates and I'm running on more than 900 000 000 log entries.
So you must have duplicate files/entries or maybe did you clean the filebeatData so that he reloads all again.

About the error. I don't mind it. Because all data still goes through. I think this appears on shutdown fo logstash/filebeat.
Maybe someone else can help you further.

[steffens (Steffen Siering)]

Filebeat uses send-at-least-once semantics. That is, upon failure filebeat has to send again. This can lead to duplicates. There is no such thing as send-exactly-once. Send-exactly-once is normally emulated by by checkpointing are some kind of deduplication (e.g. sequence numbers as used in TCP).

You can try to deduplicate by creating a message fingerprint or id (e.g. host+filename+timestamp+offset) and check if the id has been seen in the past. If you directly push to elasticsearch, setting the _id given on events contents will give you some deduplication in Elasticsearch. The action type 'create' will fail if id is already known and action type index will overwrite old document.