# Tail\_files option for filebeat not working as expected **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 25, 2023, 6:16pm UTC](https://discuss.elastic.co/t/tail-files-option-for-filebeat-not-working-as-expected/334359 "2023-05-25T18:16:04Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Puneet\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/puneet_singh/32/121459_2.png) [@Puneet\_Singh](https://discuss.elastic.co/u/Puneet_Singh)\ **Post date:** [May 25, 2023, 6:16pm UTC](https://discuss.elastic.co/t/tail-files-option-for-filebeat-not-working-as-expected/334359/1 "2023-05-25T18:16:04Z")
I have downloaded filebeat 8.8.0 today and it seems that the tail\_files option is not working as expected. When i run filebeat, the filebeat seems to be scanning entire file and scanning everything, instead of shipping only new lines. here is my configuration in filebeat.yml - ```auto # ------------------------------ Logstash Output ------------------------------- output.logstash: # The Logstash hosts hosts: ["dbsx01:5044"] # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" ``` and second change ```auto filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: filestream # Unique ID among all inputs, an ID is required. id: cadence-01 # Change to true to enable this input configuration. enabled: true tail_files: true include_lines: ['DENIED'] # Paths that should be crawled and fetched. Glob based paths. paths: - /home/puneets/sample/1.txt fields: cadence: true ``` The sample file 1.txt- ```auto 1 DENIED to ip 100.98.21.1 2 DENIED to ip 100.98.21.2 3 DENIED to ip 100.98.21.3 4 DENIED to ip 100.98.21.4 5 DENIED to ip 100.98.21.5 6 DENIED to ip 100.98.21.6 7 DENIED to ip 100.98.21.7 8 DENIED to ip 100.98.21.8 9 DENIED to ip 100.98.21.9 ........... 90 DENIED to ip 100.98.21.90 91 DENIED to ip 100.98.21.91 92 DENIED to ip 100.98.21.92 93 DENIED to ip 100.98.21.93 94 DENIED to ip 100.98.21.94 95 DENIED to ip 100.98.21.95 96 DENIED to ip 100.98.21.96 97 DENIED to ip 100.98.21.97 98 DENIED to ip 100.98.21.98 99 DENIED to ip 100.98.21.99 100 DENIED to ip 100.98.21.100 ``` here 's how i run filebeat - `./filebeat -v -e -d publisher` and here 's how i am running logstash (8.8.0)- ` ./logstash -e 'input { beats { port => 5044 } } output { stdout {} }'` when i run the filebeat , i noticed that filebeat sent the content of 1.txt from line number 21 (`21 DENIED to ip 100.98.21.21`) to line number 89 (`89 DENIED to ip 100.98.21.89`) example at logstash console - ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/6/c638bfc9b00d74ed912933be0d8ba72d96f0ba7e.png) I was expecting that nothing will appear in the logstash console output . and only if i do something like `echo "100 DENIED to ip 100.98.21.100" >> 1.txt,` only new content will appear at the logstash's stdout. Q: Can someone please help me to understand what additional configurations i need to do in the filebeat.yml to get only the latest content to be shipped to logstash? Any help / hints would be very helpful. ---
**Author:** ![Puneet\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/puneet_singh/32/121459_2.png) [@Puneet\_Singh](https://discuss.elastic.co/u/Puneet_Singh)\ **Post date:** [May 25, 2023, 6:53pm UTC](https://discuss.elastic.co/t/tail-files-option-for-filebeat-not-working-as-expected/334359/2 "2023-05-25T18:53:50Z")
UPDATE: It seems tail\_files is not supported with filestream. so i removed tail\_files and introduced ignore\_inactive in filebeat.yml as - ```auto - type: filestream # Unique ID among all inputs, an ID is required. id: cadence-01 # Change to true to enable this input configuration. enabled: true tail_files: true include_lines: ['DENIED'] # Paths that should be crawled and fetched. Glob based paths. paths: - /home/puneets/MySoftwares/FILEBEAT/8.8.0/filebeat-8.8.0-linux-x86_64/1.txt fields: cadence: true ignore_inactive: since_last_start ``` after changing the configuration ,i created a new file (1.txt) - ```auto 1 DENIED to ip 100.98.21.1 ... 100 DENIED to ip 100.98.21.100 ``` then i started filebeat and logstash as - ```auto ./filebeat -v -e -d publisher ./logstash -e 'input { beats { port => 5044 } } output { stdout {} }' ``` At this stage, there was no output on the logstash console. - which is i extected. But, when i appended new content in 1.txt every 1 second, `for _J in $(echo {101..200}); do echo "$_J DENIED to ip 100.98.21.$_J $(date)";sleep 1s; done >> 1.txt` Question1: here's the first message on logstash console - ```auto "@timestamp" => 2023-05-25T18:34:00.881Z, "message" => "106 DENIED to ip 100.98.21.106 Thu May 25 12:33:20 MDT 2023", "event" => { ``` but i was expecting following to be the first line to show up on console - `101 DENIED to ip 100.98.21.101 Thu May 25 12:33:15 MDT 2023` why did this happen? can the message show up in any order at logstash? Question 2: This was the first line - ```auto "@timestamp" => **2023-05-25T18:34:00.881Z,** "message" => "106 DENIED to ip 100.98.21.106 **Thu May 25 12:33:20 MDT** 2023", "event" => { ``` and here's the second line - "@timestamp" =\> **2023-05-25T18:34:00.881Z,** "message" =\> "114 DENIED to ip 100.98.21.114 **Thu May 25 12:33:28 MDT** 2023", "event" =\> { "original" =\> "114 DENIED to ip 100.98.21.114 Thu May 25 12:33:28 MDT 2023" Why the timestamp is same for both ? 106 and 114rth line. I was expecting a gap of ~8 seconds . Is it possible to get the correct timestamp ? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 22, 2023, 8:54pm UTC](https://discuss.elastic.co/t/tail-files-option-for-filebeat-not-working-as-expected/334359/3 "2023-06-22T20:54:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.