Filebeat keystore for basic security

Elastic Stack Beats
1

I am running Elasticsearch 7.15.2 on CentOS 8 and have it successfully taking in logs from filebeat with a clear password in the yml file.

I'm trying to get it running with a keystore, but am running into issues.

My outputs section looks as follows:
output.Elasticsearch:

  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "filebeat"
  
  # this works fine
  #password: "myclearpassword"
  
  # this is not working
  password: ${ES_PWD}

I created my keystore and added the ES_PWD entry with
sudo /usr/share/filebeat/bin/filebeat keystore create
sudo /usr/share/filebeat/bin/filebeat keystore add ES_PWD

When I restart the filebeat service I get the following log info in messages:

Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 4.
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.792Z#011INFO#011instance/beat.go:665#011Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.792Z#011INFO#011instance/beat.go:673#011Beat ID: 89b0eecf-42a7-4705-9e2b-ec7df5f221b0
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.794Z#011INFO#011[seccomp]#011seccomp/seccomp.go:124#011Syscall filter successfully installed
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.794Z#011INFO#011[beat]#011instance/beat.go:1014#011Beat info#011{"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "89b0eecf-42a7-4705-9e2b-ec7df5f221b0"}}}
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.794Z#011INFO#011[beat]#011instance/beat.go:1023#011Build info#011{"system_info": {"build": {"commit": "fd322dad6ceafec40c84df4d2a0694ea357d16cc", "libbeat": "7.15.2", "time": "2021-11-04T14:22:49.000Z", "version": "7.15.2"}}}
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.794Z#011INFO#011[beat]#011instance/beat.go:1026#011Go runtime info#011{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.16.6"}}}
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.794Z#011INFO#011[beat]#011instance/beat.go:1030#011Host info#011{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-12-07T21:52:23Z","containerized":false,"name":"ip-172-31-43-130.us-west-2.compute.internal","ip":["127.0.0.1/8","::1/128","172.31.43.130/20","fe80::7d:b4ff:feea:9f75/64"],"kernel_version":"5.4.17-2102.200.13.el8uek.x86_64","mac":["02:7d:b4:ea:9f:75"],"os":{"type":"linux","family":"","platform":"ol","name":"Oracle Linux Server","version":"8.3","major":8,"minor":3,"patch":0},"timezone":"GMT","timezone_offset_sec":0,"id":"ec2e9e2be09b264d7c61c3a0336a59e0"}}}
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.795Z#011INFO#011[beat]#011instance/beat.go:1059#011Process info#011{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 8937, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-12-08T19:26:15.860Z"}}}
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.795Z#011INFO#011instance/beat.go:309#011Setup Beat: filebeat; Version: 7.15.2
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.795Z#011INFO#011[index-management]#011idxmgmt/std.go:184#011Set output.elasticsearch.index to 'filebeat-7.15.2' as ILM is enabled.
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.795Z#011INFO#011instance/beat.go:442#011filebeat stopped.
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: 2021-12-08T19:26:16.795Z#011ERROR#011instance/beat.go:989#011Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.password' (source:'/etc/filebeat/filebeat.yml')
Dec  8 19:26:16 ip-172-31-43-130 filebeat[8937]: Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.password' (source:'/etc/filebeat/filebeat.yml')
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: filebeat.service: Main process exited, code=exited, status=1/FAILURE
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: filebeat.service: Service RestartSec=100ms expired, scheduling restart.
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: filebeat.service: Start request repeated too quickly.
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Dec  8 19:26:16 ip-172-31-43-130 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..

I think it cannot find the keystore. But I'm not certain, and not clear how to fix it so it can correctly find the keystore.

2

Yup it can't find the keystone.

Per the docs here

The proper command assuming you installed with a package manager is.

filebeat keystore create

Note I did not supply the full path when you do that it puts the key store in the wrong place in that bin directory.

Filebeat creates the keystore in the directory defined by the path.data configuration setting.

So try the command without the full path and make sure the key store ends up in the {path.home}/data directory.

3

Perfect, thanks.

I had previously tried to create the keystore from within /usr/share/filebeat using
/bin/filebeat keystore..... based on (apprarently outdated) docs I had pulled up. Then in various testing trying to figure out what was wrong had just provided the full path while in another directory.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.