FIlebeat doesn't reparse logs

Elastic Stack Beats
1

Hey,
I did read a couple of posts here but was not able to get the issue resolved.

I'm using filebeat-1.2.0(downloaded tar) and Logstash2.2.2. I want to force re-parse the files provided in the path section, but that doesn't seem to work.

I stopped file beat. I deleted the .filebeat registry. I start filebeat. I can see the paths I need appear in the registry file, but no parsing happens.

 prospectors:

paths: /var/*.log
ignore_older: 0

output:
logstash:
    # The Logstash hosts
    hosts: ["localhost:5044"]

Please let me know if i'm missing something here. Also, tail_files is commented.

Running ./filebeat -v -d * doesn't give any output

2

can you try ./filebeat -e -v -d '*'

3

Hey Steffens,
Trying that without -e didn't give any output on the console. Will try the other flag as well and post here if I find any change.

4

Also, can you thorw more light know where/what other registry files are used.
I always deleted the .filebeat and touch *.log and start filebeat again. Even this didn't allow me to re-parse files..

What I observed was, it was able to parse the directory for log( over 20k files) and get the offset info quickly

5

registry file stores file meta-data like inode and offset from last lines being published. if deleting '.filebeat' is not working, maybe filebeat is writing the registry file to another location?

6

I can see the file getting created In the same location when I restart filebeat. So the registry file does get recreated. But it doesn't somehow re-parse the log files..And only iff new data is written to one of these files will I see an event in logstash

7

Or are there multiple places for a registry file? I haven't modified the block for registry file, just using the default

8

It is interesting coz the registry file just get recreated with the offsets and inode info for a huge log collection in a matter of seconds like it knew where it last left off from

9

there is just one registry file used by filebeat. Without debug log I've no idea what's actually happening.

10

Also, can u throw more light on the ignore_older parmater. If the requirement is to re-parse logs all the time,should that be set to a very large number or 0?

Next, we want to monitor a directory which keeps getting logs files in the order of thousands per day, what should be some explicit parameters like close_older, which are good to have..

11

Let me give you repro tomorrow, after passing the debug flags. Like I had mentioned, tried using -d before and didn't see any output. Will re try and let you know..