Filebeat cannot find registry file (/var/lib/filebeat/registry/filebeat)

ntran · January 14, 2020, 1:34pm

Hi all,

I am getting this error message when I run filebeat -e:
Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory

I do not have a /var/lib/filebeat/registry/filebeat/data.json.new file, however, I do have /var/lib/filebeat/registry/filebeat/data.json

Filebeat is still harvesting files, however, how does it know it has finished harvesting the file if it cannot find the registry file?

[root@elastic-nfs01 ~]# ls -lh /var/lib/filebeat/registry/filebeat
-rw-------. 1 root root 38K Jan 14 13:23 data.json
-rwxrwxrwx. 1 root root 16 Jul 24 07:17 meta.json

Filebeat.yml
#=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log

  ignore_older: 48h
  clean_inactive: 72h
  clean_removed: true
  filebeat.registry.path: /var/lib/filebeat/registry
 # Change to true to enable this input configuration.
  # enabled: false
  paths:
    - /data/elastic-backup/logs/elastic-*/elastic-01_server.json 

- type: log

  # Change to true to enable this input configuration.
  # enabled: false
  paths:
    - /data/cluster-elastic-backup/logs/cluster-elastic*/cluster-cluster-01_server.json
  ignore_older: 48h
  clean_inactive: 72h
  clean_removed: true
  filebeat.registry.path: /var/lib/filebeat/registry
#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 1
#============================== Kibana =====================================
setup.kibana:
  host: "https://XX.XX.XX.XX:5601"
  ssl.verification_mode: "none"
#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["XX.XX.XX.XX:9200"]

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "changeme"
  ssl.verification_mode: "none"
  bulk_max_size: 50
  compression_level: 1
  worker: 2
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

Elasticsearch module

>     # Module: elasticsearch
>     # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-module-elasticsearch.html
>     - module: elasticsearch
>       # Server log
>       server:
>         enabled: false
>         # Set custom paths for the log files. If left empty,
>         # Filebeat will choose the paths depending on your OS.
>         var.paths:
>           - /data/elastic-backup/logs/elastic-hotdata01/*server.json
>     # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
>         var.convert_timezone: true
>       audit:
>         enabled: true
>         # Set custom paths for the log files. If left empty,
>         # Filebeat will choose the paths depending on your OS.
>         var.paths: 
>           - /data/elastic-backup/logs/elastic-warmdata03/elastic-01_audit*.json
>           - /data/elastic-backup/logs/elastic-hotdata01/elastic-01_audit*.log
>           -
>         # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
>         var.convert_timezone: true

shaunak · January 14, 2020, 7:29pm

Did you run this command before or after your started Filebeat?

If you restart Filebeat now, do you still get the same error? Would you mind posting the exact Filebeat logs showing the error message?

Finally, what version of Filebeat are you running?

Thanks!

ntran · January 15, 2020, 6:23am

Hi @shaunak,
Filebeat 7.2.0
I ran the command after starting Filebeat
I've restart Filebeat again and I am still getting the same error message

I ran filebeat -e and these are the logs from that.

2020-01-15T06:06:31.454Z INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.2.0

2020-01-15T06:06:31.454Z INFO [index-management] idxmgmt/std.go:178 Set output.elasticsearch.index to 'filebeat-7.2.0' as ILM is enabled.

2020-01-15T06:06:31.454Z INFO elasticsearch/client.go:166 Elasticsearch url: https://X:9200

2020-01-15T06:06:31.455Z INFO [publisher] pipeline/module.go:97 Beat name: elastic-nfs01

2020-01-15T06:06:31.456Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s

2020-01-15T06:06:31.456Z INFO instance/beat.go:421 filebeat start running.

2020-01-15T06:06:31.456Z INFO registrar/registrar.go:145 Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json

2020-01-15T06:06:31.457Z INFO registrar/registrar.go:152 States Loaded from registrar: 101

2020-01-15T06:06:31.457Z INFO crawler/crawler.go:72 Loading Inputs: 2

2020-01-15T06:06:31.462Z INFO log/input.go:148 Configured paths:

2020-01-15T06:06:31.462Z INFO input/input.go:114 Starting input of type: log; ID: X

2020-01-15T06:06:31.470Z INFO log/harvester.go:253 Harvester started for file: [file]

2020-01-15T06:06:31.492Z INFO input/input.go:114 Starting input of type: log; ID: X

2020-01-15T06:06:31.511Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:31.570Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:31.681Z INFO log/input.go:148 Configured paths:

2020-01-15T06:06:31.681Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 2

2020-01-15T06:06:31.681Z INFO cfgfile/reload.go:172 Config reloader started

2020-01-15T06:06:31.702Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:31.837Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:31.877Z INFO elasticsearch/client.go:166 Elasticsearch url: https://X:9200

2020-01-15T06:06:31.877Z WARN tlscommon/tls_config.go:79 SSL/TLS verifications disabled.

2020-01-15T06:06:32.022Z INFO elasticsearch/client.go:735 Attempting to connect to Elasticsearch version 7.5.0

2020-01-15T06:06:32.124Z INFO input/input.go:114 Starting input of type: log; ID: X

2020-01-15T06:06:32.125Z INFO cfgfile/reload.go:227 Loading of config files completed.

2020-01-15T06:06:34.569Z INFO elasticsearch/client.go:735 Attempting to connect to Elasticsearch version 7.5.0

2020-01-15T06:06:34.729Z INFO [index-management] idxmgmt/std.go:252 Auto ILM enable success.

2020-01-15T06:06:34.740Z INFO [index-management.ilm] ilm/std.go:134 do not generate ilm policy: exists=true, overwrite=false

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:265 ILM policy successfully loaded.

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:394 Set setup.template.name to '{filebeat-7.2.0 {now/d}-000001}' as ILM is enabled.

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:399 Set setup.template.pattern to 'filebeat-7.2.0-*' as ILM is enabled.

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:433 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.2.0 {now/d}-000001} as ILM is enabled.

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:437 Set settings.index.lifecycle.name in template to {filebeat-7.2.0 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.

2020-01-15T06:06:34.745Z INFO template/load.go:88 Template filebeat-7.2.0 already exists and will not be overwritten.

2020-01-15T06:06:34.745Z INFO [index-management] idxmgmt/std.go:289 Loaded index template.

2020-01-15T06:06:34.747Z INFO [index-management] idxmgmt/std.go:300 Write alias successfully generated.

2020-01-15T06:06:34.857Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(https://10.61.13.13:9200)) established

2020-01-15T06:06:34.864Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:35.622Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:35.886Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:36.248Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:37.362Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:37.514Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:38.772Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:39.331Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:39.686Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:40.618Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:41.236Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:41.471Z INFO log/harvester.go:253 Harvester started for file: /data/hrc-elastic-backup/logs/hrc-elastic-hotdata02/hrc-elastic-01_server.json

2020-01-15T06:06:42.069Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:44.581Z INFO log/harvester.go:253 Harvester started for file: /data/kca-elastic-backup/logs/kca-elastic-hotdata01/kca-cluster-01_audit.json

2020-01-15T06:06:44.939Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:45.381Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:46.348Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:49.002Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:50.904Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:51.222Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

^C2020-01-15T06:06:51.472Z INFO beater/filebeat.go:433 Stopping filebeat

2020-01-15T06:06:51.472Z INFO crawler/crawler.go:139 Stopping Crawler

2020-01-15T06:06:51.472Z INFO crawler/crawler.go:149 Stopping 2 inputs

2020-01-15T06:06:51.472Z INFO cfgfile/reload.go:230 Dynamic config reloader stopped

2020-01-15T06:06:51.472Z INFO [reload] cfgfile/list.go:118 Stopping 1 runners ...

2020-01-15T06:06:51.472Z INFO input/input.go:149 input ticker stopped

2020-01-15T06:06:51.472Z INFO input/input.go:167 Stopping Input: 974067711281631264

2020-01-15T06:06:51.472Z INFO input/input.go:149 input ticker stopped

2020-01-15T06:06:51.472Z INFO input/input.go:167 Stopping Input: 687122273758875045

ntran · January 15, 2020, 6:24am

Filebeat is currently active (running) and I can see it is harvesting logs:

Jan 15 06:23:53 elastic-nfs01 filebeat[42359]: 2020-01-15T06:23:53.431Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":12240,"time":{"ms":304}},"total":{"ticks":155340,"time":{"ms":3596},"value":155340},"user":{"ticks":143100,"time":{"ms":3292}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":90},"info":{"ephemeral_id":"X","uptime":{"ms":1050108}},"memstats":{"gc_next":90491200,"memory_alloc":78941936,"memory_total":17877918144},"runtime":{"goroutines":406}},"filebeat":{"events":{"active":1,"added":15387,"done":15386},"harvester":{"open_files":82,"running":82}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":15386,"batches":308,"total":15386},"read":{"bytes":3567224},"write":{"bytes":10595024}},"pipeline":{"clients":10,"events":{"active":4117,"published":15386,"total":15386},"queue":{"acked":15386}}},"registrar":{"states":{"current":101,"update":15386},"writes":{"success":232,"total":232}},"system":{"load":{"1":0.03,"15":0.41,"5":0.19,"norm":{"1":0.0075,"15":0.1025,"5":0.0475}}}}}}

When I also look in /var/lib/filebeat/registry/filebeat/data.json, the timestamp is today, denoting the file is being harvested by Filebeat. So I'm really confused right now in regards to what is happening.

{"source":"/data/elastic-backup/logs/elastic-warmdata03/elastic-01_audit-2019-12-15.log","offset":18049495555,"timestamp":"2020-01-15T06:29:09.344601377Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":238781751,"device":64769}}]

shaunak · January 15, 2020, 4:09pm

Are you, by any chance, running multiple Filebeat instances on the same host, which could be sharing the same registry path?

ntran · January 16, 2020, 6:36am

When I run filebeat export config, I get the following result:

As far as I am aware, I am only running one filebeat instance

filebeat:
config:
modules:
path: /etc/filebeat/modules.d/*.yml
reload:
enabled: false
inputs:

clean_inactive: 72h
clean_removed: true
filebeat:
registry:
path: /var/lib/filebeat/registry
ignore_older: 48h
paths:

/data/name-elastic-backup/logs/name-elastic-*/name-elastic-01_server.json
type: log

clean_inactive: 72h
clean_removed: true
filebeat:
registry:
path: /var/lib/filebeat/registry
ignore_older: 48h
paths:

/data/elastic-backup/logs/elastic*/cluster-01_server.json
type: log
output:
elasticsearch:
bulk_max_size: 50
compression_level: 1
hosts:

XX.XX.XX.XX:9200
password: changeme
protocol: https
ssl:
verification_mode: none
username: elastic
worker: 2
path:
config: /etc/filebeat
data: /var/lib/filebeat
home: /usr/share/filebeat
logs: /var/log/filebeat
processors:

add_host_metadata: null

add_cloud_metadata: null
setup:
kibana:
host: https://XX.XX.XX.XX:5601
ssl:
verification_mode: none
template:
settings:
index:
number_of_shards: 1

system · February 13, 2020, 6:36am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.