Filebeat configuration option 'tail_files'

Elastic Stack Beats
1

I am working on the filebeat 1.1.1. The following is the snippet of the configuation file. I assume the filebeat reads from the end of the log files as the 'tail_files' is true. But it seems the filebeat doesn't work as I thought. Any idea of this?

filebeat:
  prospectors:
    -
      paths:
        - "/var/log/nginx/*access.log"
      document_type: "nginx_access"
      exclude_files: [".tar$",".tgz$",".gz$",".bz2$",".zip$"]
      tail_files: true
      force_close_files: true
2

Can you give some more details on what is happening? What exactly does not work as expected?

Side note: I added code formatting in your post above to make it better readable

3

The access.log is 30 days old, when I start the filebeat for the first time, I notice the filebeat read all the lines of the log file. I assume the filebeat reads the only new lines that are created after the filebeat is started.

4

That is indeed not expected. Could it be that you started filebeat before without having tail_files set and that a registrar file was generated? Can you do the following?

  • Stop filebeat
  • Remove the registry file
  • Start filebeat with the -e -d "*" option

Please provide the the output from your console here.

5

Hi ruflin,
You are right, I start the filebeat with tailf_files set before.
I make a test in another server which doesn't have filebeat installed before. The filebeat read from end the of log file as I expect.
Thank you.

6

I cannot get the tail feature to work on linux - i've tried removing the .filebeat registry file and restarting with tail option = true . It doesn't read the log entries a i enter them . - I have run with the -e -d "*" options, I see a lot of "not harvesting, file didnt change" even though i'm entering lines into the log.

7

I cannot get the tail feature to work on linux - i've tried removing the .filebeat registry file and restarting with tail option = true . It doesn't read the log entries a i enter them . - I have run with the -e -d "*" options, I see a lot of "not harvesting, file didnt change" even though i'm entering lines into the log.
if i set tail : false, it reads the entire file again each time i enter a new line in my log file ( including my new line ). That is not the normal behaviour is it ? 9 to re read the entire log file ).

8

What commands do you use to write into your log file?

9

I'm just using the vi editor to enter lines in the log file.

Here is more detail -

  1. i installed filebeat-1.2.3-x86_64 from filebeat-1.2.3-x86_64.tar.gz using tar xvf cmd

  2. i entered the following in filebeat.yml
    paths :
    - /sandbox/logs/travels.log
    encoding : plain
    input_type : log
    tail : true

    logstash :
    hosts: [xxxxxx.com:5044"] ( this is the logstash instance )
    index: travels

  3. start filebeat with ./filebeat -e -d "*"

  4. enter lines into travels.log with the vi command

  5. view logstash to see if log lines make it to logstash ( they do not ).

Thanks for you help !
FF

10

additionally,
I have tried turning tail off ( tail : false )
and restarting, i see the log does pick up the initial lines, but subsequent new lines do not seem to get picked up.

  1. stopped filebeat
  2. removed .filebeat
  3. entered tail : false in filebeat.yml
  4. restart with filebeat -e -d "*"
  5. stop filebeat
  6. remove .filebeat
  7. entered tail : true in filebeat.yml
  8. restart filebeat with ./filebeat -e -d "*"
  9. enter log line via vi command

I see initial log entries are picked up. new lines ( entered using vi ) are not picked up

11

what are the steps to get a file reading from the tail - starting from initial installation ?

12

i figured out the issue - the elastic search output in filebeat.yml was uncommented - therefore it was attempting to use both elasticsearch and logstash as outputs. i commneted out the elasticsearch section and all is working fine now !

13

Glad to hear you found the solution yourself.