How to insert file,when my file is continuously updating


(prakash kumar soni) #1

Hi,
I am having 5 files, names of those files are xyz0.log ,xyz1.log,.....xyz4.log and in that file there is a text :"my name is tom".
now I am runing filebeat its only give one output to me ,but it harvest to all file and not give all five as output why?It give only one file as output.

When i change the text of my output file ,eg "my name is jerry" then it not update that output file and i not get single output.

so my question is why it not show five output ,when it harvest five file,and my second question is that how i update my file.And get output


(ruflin) #2

Filebeat does tail files, so it checks if content was append to the files. If the file length stays the same, it cannot detect changes. In your example above, I would expect filebeat to ship ry the first time you change it. Filebeat should tail all 5 files. Did you clean the registry file before "retesting"? Can you share your config file?


(prakash kumar soni) #3

yes,

filebeat:

List of prospectors to fetch data.

prospectors:

paths:
#- c:/Users/akashsoni/Downloads/k/logstash-tutorial/tutorial/.log
#- c:/Users/akashsoni/Downloads/k/logstash-tutorial/tutorial/
.err
#- c:/Users/akashsoni/Downloads/k/Test/.err
#- c:/Users/akashsoni/Downloads/Log/
.err
- c:/Users/akashsoni/Downloads/Log/*.log

input_type: log

tail_files: true

registry_file: "C:/ProgramData/filebeat/registry"

output:

logstash:

The Logstash hosts

hosts: ["localhost:5044"]

shipper:

logging:

files:

rotateeverybytes: 10485760 # = 10MB


(ruflin) #4

Why do you have tail_files enabled?


(prakash kumar soni) #5

just i am trying


(ruflin) #6

tail_files is the reason the files are not sent: https://www.elastic.co/guide/en/beats/filebeat/1.2/configuration-filebeat-options.html#_tail_files


(prakash kumar soni) #7

But,when i false the tail_files its not working ,i just change the code otherwise that line was comment


(prakash kumar soni) #8

Why first time that it give only one output,at that time my file is new.It have to show all five as output


(ruflin) #9

Filebeat persists the state so it can continue where it finished the last time. This state is persisted in the registry file. The path to the registry file is in your case registry_file: "C:/ProgramData/filebeat/registry". If you want a fresh start, remove this file.


(prakash kumar soni) #10

lets try...


(prakash kumar soni) #11

again only one output


(prakash kumar soni) #12

Its not work for me for five file it give only one output.


(ruflin) #13

Can you run filebeat with the options -e -d "*" and share the output? Or set the logging level to debug and share the log file. This should show all information needed to get further details on what is going on here.


(prakash kumar soni) #14


(prakash kumar soni) #15


(ruflin) #16

Can you please post the log files as text. It is very hard to read the screenshots and makes it impossible to search inside. Also then you can post the hole log file at once. Puting it into a gist and link it here also works.


(prakash kumar soni) #17

I am done with this problem @ruflin . Now I am facing problem of duplicate.
I have one folder of log in which there are 10 files and i get 42 hits in my elastic search.


(prakash kumar soni) #18

why it not send 10 log only to ES?


(system) #19