Filebeat cannot find registry file (/var/lib/filebeat/registry/filebeat)

Hi all,

I am getting this error message when I run filebeat -e:
Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory

I do not have a /var/lib/filebeat/registry/filebeat/data.json.new file, however, I do have /var/lib/filebeat/registry/filebeat/data.json

Filebeat is still harvesting files, however, how does it know it has finished harvesting the file if it cannot find the registry file?

[root@elastic-nfs01 ~]# ls -lh /var/lib/filebeat/registry/filebeat
-rw-------. 1 root root 38K Jan 14 13:23 data.json
-rwxrwxrwx. 1 root root 16 Jul 24 07:17 meta.json

Filebeat.yml
#=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log

  ignore_older: 48h
  clean_inactive: 72h
  clean_removed: true
  filebeat.registry.path: /var/lib/filebeat/registry
 # Change to true to enable this input configuration.
  # enabled: false
  paths:
    - /data/elastic-backup/logs/elastic-*/elastic-01_server.json 

- type: log

  # Change to true to enable this input configuration.
  # enabled: false
  paths:
    - /data/cluster-elastic-backup/logs/cluster-elastic*/cluster-cluster-01_server.json
  ignore_older: 48h
  clean_inactive: 72h
  clean_removed: true
  filebeat.registry.path: /var/lib/filebeat/registry
#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
  index.number_of_shards: 1
#============================== Kibana =====================================
setup.kibana:
  host: "https://XX.XX.XX.XX:5601"
  ssl.verification_mode: "none"
#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["XX.XX.XX.XX:9200"]

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "changeme"
  ssl.verification_mode: "none"
  bulk_max_size: 50
  compression_level: 1
  worker: 2
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
#================================ Logging =====================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

Elasticsearch module

>     # Module: elasticsearch
>     # Docs: https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-module-elasticsearch.html
>     - module: elasticsearch
>       # Server log
>       server:
>         enabled: false
>         # Set custom paths for the log files. If left empty,
>         # Filebeat will choose the paths depending on your OS.
>         var.paths:
>           - /data/elastic-backup/logs/elastic-hotdata01/*server.json
>     # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
>         var.convert_timezone: true
>       audit:
>         enabled: true
>         # Set custom paths for the log files. If left empty,
>         # Filebeat will choose the paths depending on your OS.
>         var.paths: 
>           - /data/elastic-backup/logs/elastic-warmdata03/elastic-01_audit*.json
>           - /data/elastic-backup/logs/elastic-hotdata01/elastic-01_audit*.log
>           -
>         # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
>         var.convert_timezone: true

Did you run this command before or after your started Filebeat?

If you restart Filebeat now, do you still get the same error? Would you mind posting the exact Filebeat logs showing the error message?

Finally, what version of Filebeat are you running?

Thanks!

Hi @shaunak,
Filebeat 7.2.0
I ran the command after starting Filebeat
I've restart Filebeat again and I am still getting the same error message

I ran filebeat -e and these are the logs from that.

2020-01-15T06:06:31.454Z INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.2.0

2020-01-15T06:06:31.454Z INFO [index-management] idxmgmt/std.go:178 Set output.elasticsearch.index to 'filebeat-7.2.0' as ILM is enabled.

2020-01-15T06:06:31.454Z INFO elasticsearch/client.go:166 Elasticsearch url: https://X:9200

2020-01-15T06:06:31.455Z INFO [publisher] pipeline/module.go:97 Beat name: elastic-nfs01

2020-01-15T06:06:31.456Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s

2020-01-15T06:06:31.456Z INFO instance/beat.go:421 filebeat start running.

2020-01-15T06:06:31.456Z INFO registrar/registrar.go:145 Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json

2020-01-15T06:06:31.457Z INFO registrar/registrar.go:152 States Loaded from registrar: 101

2020-01-15T06:06:31.457Z INFO crawler/crawler.go:72 Loading Inputs: 2

2020-01-15T06:06:31.462Z INFO log/input.go:148 Configured paths:

2020-01-15T06:06:31.462Z INFO input/input.go:114 Starting input of type: log; ID: X

2020-01-15T06:06:31.470Z INFO log/harvester.go:253 Harvester started for file: [file]

2020-01-15T06:06:31.492Z INFO input/input.go:114 Starting input of type: log; ID: X

2020-01-15T06:06:31.511Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:31.570Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:31.681Z INFO log/input.go:148 Configured paths:

2020-01-15T06:06:31.681Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 2

2020-01-15T06:06:31.681Z INFO cfgfile/reload.go:172 Config reloader started

2020-01-15T06:06:31.702Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:31.837Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:31.877Z INFO elasticsearch/client.go:166 Elasticsearch url: https://X:9200

2020-01-15T06:06:31.877Z WARN tlscommon/tls_config.go:79 SSL/TLS verifications disabled.

2020-01-15T06:06:32.022Z INFO elasticsearch/client.go:735 Attempting to connect to Elasticsearch version 7.5.0

2020-01-15T06:06:32.124Z INFO input/input.go:114 Starting input of type: log; ID: X

2020-01-15T06:06:32.125Z INFO cfgfile/reload.go:227 Loading of config files completed.

2020-01-15T06:06:34.569Z INFO elasticsearch/client.go:735 Attempting to connect to Elasticsearch version 7.5.0

2020-01-15T06:06:34.729Z INFO [index-management] idxmgmt/std.go:252 Auto ILM enable success.

2020-01-15T06:06:34.740Z INFO [index-management.ilm] ilm/std.go:134 do not generate ilm policy: exists=true, overwrite=false

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:265 ILM policy successfully loaded.

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:394 Set setup.template.name to '{filebeat-7.2.0 {now/d}-000001}' as ILM is enabled.

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:399 Set setup.template.pattern to 'filebeat-7.2.0-*' as ILM is enabled.

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:433 Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.2.0 {now/d}-000001} as ILM is enabled.

2020-01-15T06:06:34.740Z INFO [index-management] idxmgmt/std.go:437 Set settings.index.lifecycle.name in template to {filebeat-7.2.0 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.

2020-01-15T06:06:34.745Z INFO template/load.go:88 Template filebeat-7.2.0 already exists and will not be overwritten.

2020-01-15T06:06:34.745Z INFO [index-management] idxmgmt/std.go:289 Loaded index template.

2020-01-15T06:06:34.747Z INFO [index-management] idxmgmt/std.go:300 Write alias successfully generated.

2020-01-15T06:06:34.857Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(https://10.61.13.13:9200)) established

2020-01-15T06:06:34.864Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:35.622Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:35.886Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:36.248Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:37.362Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:37.514Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:38.772Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:39.331Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:39.686Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:40.618Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:41.236Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:41.471Z INFO log/harvester.go:253 Harvester started for file: /data/hrc-elastic-backup/logs/hrc-elastic-hotdata02/hrc-elastic-01_server.json

2020-01-15T06:06:42.069Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:44.581Z INFO log/harvester.go:253 Harvester started for file: /data/kca-elastic-backup/logs/kca-elastic-hotdata01/kca-cluster-01_audit.json

2020-01-15T06:06:44.939Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:45.381Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:46.348Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:49.002Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:50.904Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

2020-01-15T06:06:51.222Z ERROR registrar/registrar.go:374 Writing of registry returned error: rename /var/lib/filebeat/registry/filebeat/data.json.new /var/lib/filebeat/registry/filebeat/data.json: no such file or directory. Continuing...

^C2020-01-15T06:06:51.472Z INFO beater/filebeat.go:433 Stopping filebeat

2020-01-15T06:06:51.472Z INFO crawler/crawler.go:139 Stopping Crawler

2020-01-15T06:06:51.472Z INFO crawler/crawler.go:149 Stopping 2 inputs

2020-01-15T06:06:51.472Z INFO cfgfile/reload.go:230 Dynamic config reloader stopped

2020-01-15T06:06:51.472Z INFO [reload] cfgfile/list.go:118 Stopping 1 runners ...

2020-01-15T06:06:51.472Z INFO input/input.go:149 input ticker stopped

2020-01-15T06:06:51.472Z INFO input/input.go:167 Stopping Input: 974067711281631264

2020-01-15T06:06:51.472Z INFO input/input.go:149 input ticker stopped

2020-01-15T06:06:51.472Z INFO input/input.go:167 Stopping Input: 687122273758875045

Filebeat is currently active (running) and I can see it is harvesting logs:

Jan 15 06:23:53 elastic-nfs01 filebeat[42359]: 2020-01-15T06:23:53.431Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":12240,"time":{"ms":304}},"total":{"ticks":155340,"time":{"ms":3596},"value":155340},"user":{"ticks":143100,"time":{"ms":3292}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":90},"info":{"ephemeral_id":"X","uptime":{"ms":1050108}},"memstats":{"gc_next":90491200,"memory_alloc":78941936,"memory_total":17877918144},"runtime":{"goroutines":406}},"filebeat":{"events":{"active":1,"added":15387,"done":15386},"harvester":{"open_files":82,"running":82}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":15386,"batches":308,"total":15386},"read":{"bytes":3567224},"write":{"bytes":10595024}},"pipeline":{"clients":10,"events":{"active":4117,"published":15386,"total":15386},"queue":{"acked":15386}}},"registrar":{"states":{"current":101,"update":15386},"writes":{"success":232,"total":232}},"system":{"load":{"1":0.03,"15":0.41,"5":0.19,"norm":{"1":0.0075,"15":0.1025,"5":0.0475}}}}}}

When I also look in /var/lib/filebeat/registry/filebeat/data.json, the timestamp is today, denoting the file is being harvested by Filebeat. So I'm really confused right now in regards to what is happening.

{"source":"/data/elastic-backup/logs/elastic-warmdata03/elastic-01_audit-2019-12-15.log","offset":18049495555,"timestamp":"2020-01-15T06:29:09.344601377Z","ttl":-1,"type":"log","meta":null,"FileStateOS":{"inode":238781751,"device":64769}}]

Are you, by any chance, running multiple Filebeat instances on the same host, which could be sharing the same registry path?

When I run filebeat export config, I get the following result:

As far as I am aware, I am only running one filebeat instance

filebeat:
config:
modules:
path: /etc/filebeat/modules.d/*.yml
reload:
enabled: false
inputs:

  • clean_inactive: 72h
    clean_removed: true
    filebeat:
    registry:
    path: /var/lib/filebeat/registry
    ignore_older: 48h
    paths:
    • /data/name-elastic-backup/logs/name-elastic-*/name-elastic-01_server.json
      type: log
  • clean_inactive: 72h
    clean_removed: true
    filebeat:
    registry:
    path: /var/lib/filebeat/registry
    ignore_older: 48h
    paths:
    • /data/elastic-backup/logs/elastic*/cluster-01_server.json
      type: log
      output:
      elasticsearch:
      bulk_max_size: 50
      compression_level: 1
      hosts:
    • XX.XX.XX.XX:9200
      password: changeme
      protocol: https
      ssl:
      verification_mode: none
      username: elastic
      worker: 2
      path:
      config: /etc/filebeat
      data: /var/lib/filebeat
      home: /usr/share/filebeat
      logs: /var/log/filebeat
      processors:
  • add_host_metadata: null
  • add_cloud_metadata: null
    setup:
    kibana:
    host: https://XX.XX.XX.XX:5601
    ssl:
    verification_mode: none
    template:
    settings:
    index:
    number_of_shards: 1