Manual Edit of Registry file

Elastic Stack Beats
1

I have manually edited the registry file on one of my filebeats containers in an attempt to get it to reprocess specific files. It will load the edited registry, and the logs indicate that it no longer recognizes the the files I have removed from it's list, but it still will not reprocess these files. Is there another location that I need to update to get it to reprocess these files?

Deleting the registry completely and reprocessing everything is not an option, I have thousands of files that equate to billions of entries into elastic.

2

What kind of manul edit did you do? Did you just remove entries from the JSON, or did you modify the offset?

Can you share your filebeat configuration? Have you configured some setting to ignore older files?

3

I removed the entries from the JSON entirely that matched a specific pattern. These are the types of entries that I was removing:

{
	"FileStateOS": {
		"device": 64783,
		"inode": 26169086743
	},
	"timestamp": "2018-10-03T21:22:46.5256349Z",
	"offset": 43047599,
	"source": "/data/some_datagroup/some_datafile.txt",
	"meta": null,
	"ttl": -1,
	"type": "log"
}

Here is my filebeats.yml file, I'm using filebeats 6.4 within docker:

filebeat.inputs:
- type: log
  processors:
    - drop_fields:
        fields: ["prospector","input","offset"]

  enabled: true
  paths:
    - /data/*/*.txt
  json.keys_under_root: true

processors:
 - drop_fields:
     fields: ["beat", "host"]

setup.template.enabled: false
output.logstash:
  hosts: ["logstash:5044"]
4

If the files (by inode) still exist, I don't see why they should not be collected.

Can you check if the files do appear in the filebeat log file.

When stopping filebeat, do the files appear in the registry file? If not, they might not be available anymore.

5

Files are still available, they do not appear in the filebeat log. They are not in the registry either.

Additionally, the filebeat container also generates an additional registry file called registry.new when I make these changes. It doesn't make a whole lot of sense to me :confused:

6

Any other kind of errors?

When writing the registry file, filebeat first creates the file registry.new, with all new contents. Right after serializing to registry.new, filebeat uses a replace operation to atomically replace the old registry file with registry.new. The fact that you still have the registry.new around might point to an IO error. Have you change file credentials or permissions by accident when editing?

7

I will take a look if it has something to do with the registry.new file... that might be a weird artifact from docker volumes. I know that when I slice the entries out the filebeat registry parse at startup indicates it's tracking the correct amount of files though, so not sure if that's where the issue is...

I might try the offset thing as well. Thanks for the ideas.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.