Logstash doesn't read changes in file and need to start from bin every time

Elastic Stack Logstash
1

Hello,

I have installed Logstash 6.6.1 in rhel 7, from RPM.
Now I have created a file under /etc/logstash/conf.d/apache.conf . This file keeps updating itself.
I ran following command to send logs to elasticsearch:
./logstash -f /etc/logstash/conf.d/apache.conf

It works fine, i can see the logs till command run time. However next time the file is updated, i don't see any changes in elasticsearch.
I have to run above command every time to see updated logs in elasticsearch.
Is there any way to automatic reload logstash, so that i can see updated logs?

2

Logstash tails files so as long as new data is appended to the file it should be read. If existing content however is modified this is unlikely to get picked up. What type of data does the file contain? How is data updated?

3

this file contains HTTPD access logs, i am using rsync to sync logs from a cloud server, as my elasticsearch server is not public.here is the content of file:

`input
{
file {
path => "/root/Desktop/access_log"
type => "logs"
start_position => "beginning"
}
}
filter
{
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
mutate {
convert => {"bytes" => "integer" }
}
date{
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
locale => en
remove_field => "timestamp"
}
geoip {
source => "clientip"
}
useragent {
source => "agent"
target => "useragent"
}
}
output{
elasticsearch {
hosts => "http://localhost:9200"
index => "mysite"
document_type => "usagereport"
}

stdout {}

}`

4

Using raunchy to update files is not recommended and I would expect you to end up with lots of duplicates as each update would give a new inode which Filebeat would interpret as a new file and read from the beginning.

5

Hi,

When I re-run same logstash command I do not get any duplicates, its just how logstash will read updated file? or whenever changes are being done on file. I am not using Filebeat right now.

6

Logstash file input as far as I know behaves the same way, so I would expect similar behaviour. Are you rsyncing the full and expanding file each time?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.