Filebeat can't read log file continuously

[mgazanayi (Marouane)]

Hello,
I'm trying to continuously read a log file, and output it to an other file. (In reality, I'm trying to isolate why Filebeat only reads the log file once).

Here is my Filebeat configuration:

filebeat.inputs:
- type: log 
  enabled: true
  paths:
    - /var/log/standalone/query.log

output.file:
  path: "/var/log/standalone/output"
  filename: filebeat.log

I run it in a docker container mapping the /var/log/standalone on a local directory in the host machine.

When I start my container, the first time everything is ok, but even if the query.log is updated, nothing is happening at all.

When I list the opened files (after Filebeat starts):

COMMAND     PID      USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
com.docke  2291 mgazanayi  213r   REG    1,4   210029 8635149898 /Users/mgazanayi/neo4j/standalone/logs/query.log
java      84737 mgazanayi  260w   REG    1,4   210029 8635149898 /Users/mgazanayi/neo4j/standalone/logs/query.log
tail      86001 mgazanayi    3r   REG    1,4   210029 8635149898 /Users/mgazanayi/neo4j/standalone/logs/query.log

When the query.log is updated, tail is showing it also to the stdout, but nothing is updated in the filebeat.out file.

But once I stop my app writing the logs (Neo4j):

COMMAND     PID      USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
com.docke  2291 mgazanayi  213r   REG    1,4   278303 8635149898 /Users/mgazanayi/neo4j/standalone/logs/query.log
tail      86001 mgazanayi    3r   REG    1,4   278303 8635149898 /Users/mgazanayi/neo4j/standalone/logs/query.log

Everything got updated to the file.

I reproduced the same problem with an input file which is created on the fly with tail, what I do, is I tail the query.log file to a new file, and then read from this newly created file. Same thing.

EDIT:

I inserted manually echo {\"test\":\"test\"} >> query.log into the log file, and that time, all the changes got detected.

Any idea please?
Thanks

[jsoriano (Jaime Soriano)]

Hey @mgazanayi,

How do you write this file? Do you append to it, or do you overwrite it continuously?

Filebeat is designed to read logs from files whose content is being appended, as uses to happen with log files, and happens with your test using >> query.log.

I would recommend to append always to the same file, and rotate it when it gets to certain size.