fileBeats failing

[NewmazN24 (Steveb)]

Hey guys,

My environment - Dev

Master nodes (Elasticsearch & Logstash are installed) x 2
Kibana node (Only Kibana) x 1
All m servers are on CentOS7

Before you ask

I can reach my master nodes
Both nodes see eachother
I can access Kibana

What I am trying to achieve
Having syslog from Elasticsearch servers sent to my ELK. Why ? To understand how it works and the output from my configuration.

That being said, launching logstash works, but not filebeat.

Config files
Logstash conf
Path: /etc/logstash/conf.d/logstash-syslog.conf

input {
beats { port => "5044" }
}

filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}

}
output {
elasticsearch {
hosts => ["192.168.131.134:9200","192.168.131.136:9200"]
}
stdout {
codec => rubydebug
}
}

Pipeline conf

    pipeline.id: filebeat
    queue.type: persisted
    path.config: "/etc/logstash/conf.d/logstash-syslog.conf"

Filebeat conf

filebeat.input:

 -  input_type: log
    enabled: true
    paths:
        /var/log/messages
        /var/log/*.log
        /var/log/syslog

setup.kibana:
host: "192.168.131.135:5601"

output.logstash:
hosts: ["192.168.131.134:5044", "192.168.131.136:5044"]

Error facing

Sep 05 15:32:06 elk-prod-1 systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE
Sep 05 15:32:06 elk-prod-1 systemd[1]: Unit filebeat.service entered failed state.
Sep 05 15:32:06 elk-prod-1 systemd[1]: filebeat.service failed.
Sep 05 15:32:06 elk-prod-1 systemd[1]: filebeat.service holdoff time over, scheduling restart.
Sep 05 15:32:06 elk-prod-1 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Sep 05 15:32:06 elk-prod-1 systemd[1]: start request repeated too quickly for filebeat.service
Sep 05 15:32:06 elk-prod-1 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to E...rch..
Sep 05 15:32:06 elk-prod-1 systemd[1]: Unit filebeat.service entered failed state.
Sep 05 15:32:06 elk-prod-1 systemd[1]: filebeat.service failed.

-- The start-up result is done.
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.173-0400 INFO instance/beat.go:606 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.174-0400 INFO instance/beat.go:614 Beat ID: 527d101c-82a0-4aa0-b6f2-27ec8b298703
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.174-0400 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.174-0400 INFO [beat] instance/beat.go:902 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "527d101c-82a0-4aa0-b6f2-27ec8b298703"}}}
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.174-0400 INFO [beat] instance/beat.go:911 Build info {"system_info": {"build": {"commit": "a4be71b90ce3e3b8213b616adfcd9e455513da45", "libbeat": "7.3.1", "time": "2019-08-19T19:30:50.000Z", "version": "7.3.1"}}}
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.174-0400 INFO [beat] instance/beat.go:914 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.12.4"}}}
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.176-0400 INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.3.1
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.177-0400 INFO [publisher] pipeline/module.go:97 Beat name: elk-prod-1
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.177-0400 INFO instance/beat.go:385 filebeat stopped.
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: 2019-09-05T14:46:44.177-0400 ERROR instance/beat.go:877 Exiting: no modules or inputs enabled and configuration reloading disabled. What files do you want me to watch?
Sep 05 14:46:44 elk-prod-1 filebeat[2150]: Exiting: no modules or inputs enabled and configuration reloading disabled. What files do you want me to watch?
Sep 05 14:46:44 elk-prod-1 systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE
Sep 05 14:46:44 elk-prod-1 systemd[1]: Unit filebeat.service entered failed state.
Sep 05 14:46:44 elk-prod-1 systemd[1]: filebeat.service failed.
Sep 05 14:46:44 elk-prod-1 systemd[1]: filebeat.service holdoff time over, scheduling restart.
Sep 05 14:46:44 elk-prod-1 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..

[NewmazN24 (Steveb)]

Hello folks,

• I have tried to open ports from firewall-cmd on both TCP/UDP since I read that syslog may be using it.
• I removed the second node from Filebeat.conf & Logstash.conf, so it only has the .136.
• Tried to put filebeat.prospectors instead of filebeat.input.
• Tried - input_type and - type.

For some reason, when I try to run filebeat modules enable logstash, is doesn't work. I will post the error later on today.