Tail_files option for filebeat not working as expected

Elastic Stack Beats
1

I have downloaded filebeat 8.8.0 today and it seems that the tail_files option is not working as expected. When i run filebeat, the filebeat seems to be scanning entire file and scanning everything, instead of shipping only new lines.

here is my configuration in filebeat.yml -

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["dbsx01:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

and second change

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: cadence-01

  # Change to true to enable this input configuration.
  enabled: true
  tail_files: true
  include_lines: ['DENIED']
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /home/puneets/sample/1.txt
  fields:
    cadence: true

The sample file 1.txt-

1 DENIED to ip 100.98.21.1
2 DENIED to ip 100.98.21.2
3 DENIED to ip 100.98.21.3
4 DENIED to ip 100.98.21.4
5 DENIED to ip 100.98.21.5
6 DENIED to ip 100.98.21.6
7 DENIED to ip 100.98.21.7
8 DENIED to ip 100.98.21.8
9 DENIED to ip 100.98.21.9
...........
90 DENIED to ip 100.98.21.90
91 DENIED to ip 100.98.21.91
92 DENIED to ip 100.98.21.92
93 DENIED to ip 100.98.21.93
94 DENIED to ip 100.98.21.94
95 DENIED to ip 100.98.21.95
96 DENIED to ip 100.98.21.96
97 DENIED to ip 100.98.21.97
98 DENIED to ip 100.98.21.98
99 DENIED to ip 100.98.21.99
100 DENIED to ip 100.98.21.100

here 's how i run filebeat -
./filebeat -v -e -d publisher
and here 's how i am running logstash (8.8.0)-
./logstash -e 'input { beats { port => 5044 } } output { stdout {} }'

when i run the filebeat , i noticed that filebeat sent the content of 1.txt
from line number 21 (21 DENIED to ip 100.98.21.21) to line number 89 (89 DENIED to ip 100.98.21.89)
example at logstash console -

image
image1393×653 18.1 KB

I was expecting that nothing will appear in the logstash console output .
and only if i do something like

echo "100 DENIED to ip 100.98.21.100" >> 1.txt,
only new content will appear at the logstash's stdout.

Q: Can someone please help me to understand what additional configurations i need to do in the filebeat.yml to get only the latest content to be shipped to logstash?
Any help / hints would be very helpful.

2

UPDATE:
It seems tail_files is not supported with filestream.
so i removed tail_files and introduced ignore_inactive in filebeat.yml as -

- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: cadence-01

  # Change to true to enable this input configuration.
  enabled: true
  tail_files: true
  include_lines: ['DENIED']
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /home/puneets/MySoftwares/FILEBEAT/8.8.0/filebeat-8.8.0-linux-x86_64/1.txt
  fields:
    cadence: true
  ignore_inactive: since_last_start

after changing the configuration ,i created a new file (1.txt) -

1 DENIED to ip 100.98.21.1
...
100 DENIED to ip 100.98.21.100

then i started filebeat and logstash as -

./filebeat -v -e -d publisher
./logstash -e 'input { beats { port => 5044 } } output { stdout {} }'

At this stage, there was no output on the logstash console. - which is i extected.

But, when i appended new content in 1.txt every 1 second,
for _J in $(echo {101..200}); do echo "$_J DENIED to ip 100.98.21.$_J $(date)";sleep 1s; done >> 1.txt

Question1:
here's the first message on logstash console -

    "@timestamp" => 2023-05-25T18:34:00.881Z,
       "message" => "106 DENIED to ip 100.98.21.106 Thu May 25 12:33:20 MDT 2023",
         "event" => {

but i was expecting following to be the first line to show up on console -
101 DENIED to ip 100.98.21.101 Thu May 25 12:33:15 MDT 2023

why did this happen? can the message show up in any order at logstash?

Question 2:
This was the first line -

    "@timestamp" => **2023-05-25T18:34:00.881Z,**
       "message" => "106 DENIED to ip 100.98.21.106 **Thu May 25 12:33:20 MDT** 2023",
         "event" => {

and here's the second line -
"@timestamp" => 2023-05-25T18:34:00.881Z,
"message" => "114 DENIED to ip 100.98.21.114 Thu May 25 12:33:28 MDT 2023",
"event" => {
"original" => "114 DENIED to ip 100.98.21.114 Thu May 25 12:33:28 MDT 2023"

Why the timestamp is same for both ? 106 and 114rth line.
I was expecting a gap of ~8 seconds . Is it possible to get the correct timestamp ?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.