Filebeat Registry Issue

Elastic Stack Beats
1

I am using filbeat version 7.2.0. Using Docker . I am succesfully creating image and running iamge without any issues. But I am having issues on registry file. I specified the registry file to be /etc/filebeat/data/registry. Not Sure why it saying No registry file not Found. And it creating a registry file but with this path /etc/filebeat/filebeat/data/registry/meta.json&&data.json.
No registry home found. Create: /etc/filebeat/data/registry/filebeat
INFO registrar/migrate.go:112 Initialize registry meta file

Could you please help with above issue. Where i can use registry file under /etc/filebeat/data/registry.

Appreciated your help

2

In 7.0+ the setting for the path is filebeat.registry.path. The registry is no single file, but a directory. Currently this directory contains meta.json and data.json files. The meta.json file currently contains a version number, allowing beats to migrate schemas in the future + data.json is the actual data in json format.

3

Hi @steffens thanks a lot for info. That is very useful info. I am in process of updating filebeat version from 6.2.4 to 7.2. I have two questions can you please help me

  1. So in my filebeat.yml file I need to Specify as filebeat.registry.path: /etc/filebeat/data/registry/filebeat right?

2.Also doing filebat as docker and using beanstalk to deploy my image. I am facing some issues on volumes mounting filebeat from source to container path
In past i mounted to /etc/filebeat/data for 6.2.4 version, do i need to change it to /etc/filebeat/data/registry/filebeat?

please advise i appreciate your help here a lot

4

So in my filebeat.yml file I need to Specify as filebeat.registry.path: /etc/filebeat/data/registry/filebeat right?

The default value if filebeat.registry.path: is ${path.data}/registry. Inside this directory filebeat will create the named registry filebeat, with all it's files. In your case the setting should be:

filebeat.registry.path: /etc/filebeat/data/registry

If filebeat finds an old registry file at /etc/filebeat/data/registry, then it will copy the file into /etc/filebeat/data/registry/filebeat/data.json and create a file named /etc/filebeat/data/registry/filebeat/meta.json with contents {"version": "0"}.

Also doing filebat as docker and using beanstalk to deploy my image. I am facing some issues on volumes mounting filebeat from source to container path
In past i mounted to /etc/filebeat/data for 6.2.4 version, do i need to change it to /etc/filebeat/data/registry/filebeat?

Don't change the mount. Beats configure a path.data setting where to store internal data for reuse. For example /etc/filebeat/data/meta.json. Starting with 7.0 filebeat will create a sub-directory for it's registry, but the data path didn't change.

5

@steffens thanks a lot for your response. I tried as suggested. But seems no luck. I still don't see data reporting to Logstash. I trieed to see connection issues, config issues etc. Didn't didn't find anything.
The only error i see is
No registry home found. Create: /etc/filebeat/data/registry/filebeat
2019-08-16T15:20:46.988Z INFO registrar/migrate.go:112 Initialize registry meta file
2019-08-16T15:20:46.989Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2019-08-16T15:20:46.991Z INFO registrar/registrar.go:108 No registry file found under: /etc/filebeat/data/registry/filebeat/data.json. Creating a new registry file.

Not usre why it saying No registry home found. Could you help me here please
Thanks a lot for your help here

6

This log message is no error. It is just some informational message telling you that the filebeat registry does not exists yet and a new one will be generated.

Are you sure about the /etc/filebeat/data path to be correct? What does your docker config look like?

What does your filebeat config look like?

Have you checked with netstat (or ss), if there is a connection between beats and logstash at all?

7

@steffens Thanks a lot for your time. Not sure waht happen but after 24 hours i see data reporting to filebeat. Does it take time to report data to kibana when we do version update?

I do some questions,

  1. IS there a way where i can remove some fields on kibana?
  2. How to know table size of my logs?

TIA.

8

Not having any configs, I can just guess. I guess this is due to your index and template configuration. The events in 6.2 and 7.0 are not compatible due to the introduction of ECS (Elastic Common Schema). Normally in Beats we version the template and index names, so to not run into mapping conflicts and such. Thanks to daily indices, you might have started to write into a new index, that was not at conflict.

9

@steffens. Thanks for your response. that is very useful to me.
I Used filebeat.template.json from 6.2. for 7.2. Do we have to use a new template for 7.0? Or No need to have template file for 7.0? If needed where can i find this?
TIA

10

@steffens Any Updates please. Thanks for your time

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.