How to insert file,when my file is continuously updating

Elastic Stack Beats
1

Hi,
I am having 5 files, names of those files are xyz0.log ,xyz1.log,.....xyz4.log and in that file there is a text :"my name is tom".
now I am runing filebeat its only give one output to me ,but it harvest to all file and not give all five as output why?It give only one file as output.

When i change the text of my output file ,eg "my name is jerry" then it not update that output file and i not get single output.

so my question is why it not show five output ,when it harvest five file,and my second question is that how i update my file.And get output

2

Filebeat does tail files, so it checks if content was append to the files. If the file length stays the same, it cannot detect changes. In your example above, I would expect filebeat to ship ry the first time you change it. Filebeat should tail all 5 files. Did you clean the registry file before "retesting"? Can you share your config file?

3

yes,

filebeat:

List of prospectors to fetch data.

prospectors:

paths:
#- c:/Users/akashsoni/Downloads/k/logstash-tutorial/tutorial/.log
#- c:/Users/akashsoni/Downloads/k/logstash-tutorial/tutorial/.err
#- c:/Users/akashsoni/Downloads/k/Test/.err
#- c:/Users/akashsoni/Downloads/Log/.err
- c:/Users/akashsoni/Downloads/Log/*.log

input_type: log

tail_files: true

registry_file: "C:/ProgramData/filebeat/registry"

output:

logstash:

The Logstash hosts

hosts: ["localhost:5044"]

shipper:

logging:

files:

rotateeverybytes: 10485760 # = 10MB

4

Why do you have tail_files enabled?

5

just i am trying

6

tail_files is the reason the files are not sent: https://www.elastic.co/guide/en/beats/filebeat/1.2/configuration-filebeat-options.html#_tail_files

7

But,when i false the tail_files its not working ,i just change the code otherwise that line was comment

8

Why first time that it give only one output,at that time my file is new.It have to show all five as output

9

Filebeat persists the state so it can continue where it finished the last time. This state is persisted in the registry file. The path to the registry file is in your case registry_file: "C:/ProgramData/filebeat/registry". If you want a fresh start, remove this file.

10

lets try...

11

again only one output

12

Its not work for me for five file it give only one output.

13

Can you run filebeat with the options -e -d "*" and share the output? Or set the logging level to debug and share the log file. This should show all information needed to get further details on what is going on here.

14

Untitled.jpg857×715 337 KB

15

Untitled2.jpg872×568 257 KB

16

Can you please post the log files as text. It is very hard to read the screenshots and makes it impossible to search inside. Also then you can post the hole log file at once. Puting it into a gist and link it here also works.

17

I am done with this problem @ruflin . Now I am facing problem of duplicate.
I have one folder of log in which there are 10 files and i get 42 hits in my elastic search.

18

why it not send 10 log only to ES?