Hi,
I'm running Logstash and Filebeat with what I believe to be simple configurations. I have two syslog listeners, one on UDP 9001 and one on UDP 9002.
When running Logstash as a service and when sending logs to 9001, I see them in Kibana. When sending logs to 9002, I do not see them in Kibana.
What I find very strange is that if I run Logstash from the cli with
/usr/share/logstash/bin/logstash --debug -f /etc/logstash/conf.d/40-elasticsearch-output.conf
I see logs coming in in the debug logs, and all logs from 9001 and 9002 appear in Kibana.
If I ctl-c the debug process and start the service, some logs from 9002 suddenly come in, but after some time, they stop coming in.
Netstat shows 9001 and 9002 are listening and when sending logs to 9002 I always see them coming in using tcpdump
At the moment I am completely at a loss to troubleshoot this or explain what is happening. I've never played with the ELK stack before and I'm going from the docs and blog articles I've found.
Any help is much appreciated.
Versions
Logstash: logstash/stable,now 1:7.17.1-1 amd64
Filebeat: filebeat/stable,now 7.17.1 amd64
Configurations
filebeat.yml
- type: syslog
enabled: true
format: auto
protocol.udp:
host: "1.1.1.1.148:9001" # not real ip
tags: ["aruba-cxos", "aruba"]
fields:
log_type: "aruba-cxos"
- type: syslog
enabled: true
format: auto
protocol.udp:
host: "1.1.1.1:9002"
tags: ["aruba-switchos", "aruba"]
fields:
log_type: "aruba-switchos"
#### skipping
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["0.0.0.0:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
40-Elasticsearch-output.conf
input {
beats {
port => 5044
}
}
output {
if [@metadata][pipeline] {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
pipeline => "%{[@metadata][pipeline]}"
}
} else {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
}
Netstat shows 9001 and 9002 are listening and when sending logs to 9002 I always see them coming in using tcpdump