Failed to start Filebeat MISP treats

Hi Everyone im glad to be part of this comunity. Im here beacause a got a error message when I try to start the filebeat service after do some modifications related to a enable de MISP threat intel Logs.
Here is the filebeat.yml

# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
        hosts: ["172.X0.1XX.12:9200"]
   Protocol - either `http` (default) or `https`.
  #protocol: "https"
  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
        username: "elastic"
        password: "VwIL2s3kJre9HNBE9A5u"
#setup.kibana:
#       host: "<172.X0.1XX.12>"
# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["172.X0.1XX.12:5044"]
  username: "logstash_system"
  password: "4NnCxvCwnoY8YRcQKNqc"
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

the issue stat when a modifiy this part of the file

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
        hosts: ["172.X0.1XX.12:9200"]

   Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
        username: "elastic"
        password: "VwIL2s3kJre9HNBE9A5u"
#setup.kibana:
        host: "<172.X0.1XX.12>"

Journalctl -xe -u filebeat:

░░ A stop job for unit filebeat.service has finished.
░░
░░ The job identifier is 8696 and the job result is done.
Apr 22 13:32:36 debian-suricata systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
░░ Subject: A start job for unit filebeat.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit filebeat.service has finished successfully.
░░
░░ The job identifier is 8696.
Apr 22 13:32:36 debian-suricata filebeat[12910]: Exiting: error loading config file: yaml: line 135: found character that cannot start any token
Apr 22 13:32:36 debian-suricata systemd[1]: filebeat.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit filebeat.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 22 13:32:36 debian-suricata systemd[1]: filebeat.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit filebeat.service has entered the 'failed' state with result 'exit-code'.
Apr 22 13:32:36 debian-suricata systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ Automatic restarting of the unit filebeat.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Apr 22 13:32:36 debian-suricata systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
░░ Subject: A stop job for unit filebeat.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A stop job for unit filebeat.service has finished.
░░
░░ The job identifier is 8766 and the job result is done.
Apr 22 13:32:36 debian-suricata systemd[1]: filebeat.service: Start request repeated too quickly.
Apr 22 13:32:36 debian-suricata systemd[1]: filebeat.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit filebeat.service has entered the 'failed' state with result 'exit-code'.
Apr 22 13:32:36 debian-suricata systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
░░ Subject: A start job for unit filebeat.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit filebeat.service has finished with a failure.
░░
░░ The job identifier is 8766 and the job result is failed.

Log filebeat

root@debian-suricata:/var/log/filebeat# cat filebeat
2022-04-22T13:09:07.174-0400    INFO    instance/beat.go:685    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-04-22T13:09:07.175-0400    INFO    instance/beat.go:693    Beat ID: 6735e9d4-e7d6-469e-8126-5152dce51547
2022-04-22T13:09:10.179-0400    WARN    [add_cloud_metadata]    add_cloud_metadata/provider_aws_ec2.go:79       read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.
2022-04-22T13:09:10.180-0400    INFO    [beat]  instance/beat.go:1039   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "6735e9d4-e7d6-469e-8126-5152dce51547"}}}
2022-04-22T13:09:10.180-0400    INFO    [beat]  instance/beat.go:1048   Build info      {"system_info": {"build": {"commit": "f6042bc3407cc10201cfd8c7574d8b0a88a699db", "libbeat": "7.17.2", "time": "2022-03-28T09:47:58.000Z", "version": "7.17.2"}}}
2022-04-22T13:09:10.180-0400    INFO    [beat]  instance/beat.go:1051   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.17.6"}}}
2022-04-22T13:09:10.180-0400    INFO    [beat]  instance/beat.go:1055   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-04-20T10:51:12-04:00","containerized":false,"name":"debian-suricata","ip":["127.0.0.1/8","::1/128","172.X0.1XX.12/24","fe80::649e:51ff:feb0:46c3/64"],"kernel_version":"5.10.0-13-amd64","mac":["66:9e:51:b0:46:c3"],"os":{"type":"linux","family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"11 (bullseye)","major":11,"minor":0,"patch":0,"codename":"bullseye"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"8925b48d41884fb8937408a0a53495ee"}}}
2022-04-22T13:09:10.181-0400    INFO    [beat]  instance/beat.go:1084   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/home/itsupport", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 12664, "ppid": 12662, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2022-04-22T13:09:06.670-0400"}}}
2022-04-22T13:09:10.183-0400    INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.17.2
2022-04-22T13:09:10.183-0400    INFO    [publisher]     pipeline/module.go:113  Beat name: debian-suricata
2022-04-22T13:09:10.185-0400    WARN    beater/filebeat.go:202  Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-04-22T13:09:10.185-0400    ERROR   instance/beat.go:1014   Exiting: Index management requested but the Elasticsearch output is not configured/enabled

Regards and thanks for any help

Hi @Tetsuho, welcome to the Elastic community.

Could you please confirm that you have the following at line 135 of filebeat.yml?

Protocol - either `http` (default) or `https`.`

This line is intended to by a comment and should be prefixed by # (the line itself is not valid yaml)

# Protocol - either `http` (default) or `https`.`

If that's not the problem please review your yaml files and assure they contain valid yaml.

Please, be aware that you shared in clear text your password credentials: I'd urge you to rotate them

Hi Andrea thanks so very much for you answer, the passwords exposed killed my hopes to fix that installation. So I did a fresh install and it look to be working fine.
I got a issue now trying to use filebeat and misp threat intel.

I must to create a new topic for that ?

Regards

Hello @Tetsuho ,

glad the new installation is working properly

What is the question you have about filebeat and misp threat intel?

Hi Andrea, I'm trying to enable that module following the instructions:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["172.20.188.13:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "intentionally deleted"
#modificacion solicitada para activar el modulo MISP
  setup.kibana:
  host: "<172.X0.1XX.1X:5601>"
root@debian-suricata:/home/itsupport# filebeat modules enable misp
Module misp is already enabled
root@debian-suricata:/home/itsupport# filebeat setup --modules=misp -e --dashboards
2022-04-28T12:02:37.441-0400    INFO    instance/beat.go:685    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-04-28T12:02:37.442-0400    INFO    instance/beat.go:693    Beat ID: 534056f3-17c5-4e34-9ff2-ae79ac5829c6
2022-04-28T12:02:40.444-0400    WARN    [add_cloud_metadata]    add_cloud_metadata/provider_aws_ec2.go:79       read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/latest/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.
2022-04-28T12:02:40.445-0400    INFO    [beat]  instance/beat.go:1039   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "534056f3-17c5-4e34-9ff2-ae79ac5829c6"}}}
2022-04-28T12:02:40.445-0400    INFO    [beat]  instance/beat.go:1048   Build info      {"system_info": {"build": {"commit": "1993ee88a11cb34f61a1fb45c7c3cf50533682cb", "libbeat": "7.17.3", "time": "2022-04-19T09:27:20.000Z", "version": "7.17.3"}}}
2022-04-28T12:02:40.445-0400    INFO    [beat]  instance/beat.go:1051   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.17.8"}}}
2022-04-28T12:02:40.445-0400    INFO    [beat]  instance/beat.go:1055   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-04-25T14:21:13-04:00","containerized":false,"name":"debian-suricata","ip":["127.0.0.1/8","::1/128","172.X0.1XX.12/24","fe80::a803:eeff:fe0f:ed98/64"],"kernel_version":"5.10.0-13-amd64","mac":["aa:03:ee:0f:ed:98"],"os":{"type":"linux","family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"11 (bullseye)","major":11,"minor":0,"patch":0,"codename":"bullseye"},"timezone":"-04","timezone_offset_sec":-14400,"id":"191e8d75c15d418684a96b7681a70883"}}}
2022-04-28T12:02:40.446-0400    INFO    [beat]  instance/beat.go:1084   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/home/itsupport", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 5983, "ppid": 5944, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2022-04-28T12:02:36.270-0400"}}}
2022-04-28T12:02:40.446-0400    INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.17.3
2022-04-28T12:02:40.446-0400    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.17.3' as ILM is enabled.
2022-04-28T12:02:40.446-0400    INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: http://172.X0.1XX.13:9200
2022-04-28T12:02:40.447-0400    INFO    [publisher]     pipeline/module.go:113  Beat name: debian-suricata
2022-04-28T12:02:40.448-0400    INFO    beater/filebeat.go:118  Enabled modules/filesets: misp (threat)
2022-04-28T12:02:40.449-0400    WARN    beater/filebeat.go:136  Fileset `threat` for module `misp` is loaded but was not explicitly defined in the config. Starting from v8.0 this fileset won't be loaded unless explicitly defined.
Loading dashboards (Kibana must be running and reachable)
2022-04-28T12:02:40.450-0400    INFO    kibana/client.go:180    Kibana url: http://172.X0.1XX.13:5601
2022-04-28T12:02:43.363-0400    INFO    kibana/client.go:180    Kibana url: http://172.X0.1XX.13:5601
2022-04-28T12:02:43.447-0400    INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:101    add_cloud_metadata: hosting provider type not detected.

I see this line advising about a mismatch configuration but I don't understand what does it mean

beater/filebeat.go:136  Fileset `threat` for module `misp` is loaded but was not explicitly defined in the config. Starting from v8.0 this fileset won't be loaded unless explicitly defined.

Thanks for all your help Andrea

Hi,

the misp module is deprecated in 7.14.0 , use the threatintel module instead.