Logstash working from cli, but not whilst running as a service

Hi,

I'm running Logstash and Filebeat with what I believe to be simple configurations. I have two syslog listeners, one on UDP 9001 and one on UDP 9002.

When running Logstash as a service and when sending logs to 9001, I see them in Kibana. When sending logs to 9002, I do not see them in Kibana.

What I find very strange is that if I run Logstash from the cli with
/usr/share/logstash/bin/logstash --debug -f /etc/logstash/conf.d/40-elasticsearch-output.conf

I see logs coming in in the debug logs, and all logs from 9001 and 9002 appear in Kibana.
If I ctl-c the debug process and start the service, some logs from 9002 suddenly come in, but after some time, they stop coming in.

Netstat shows 9001 and 9002 are listening and when sending logs to 9002 I always see them coming in using tcpdump

At the moment I am completely at a loss to troubleshoot this or explain what is happening. I've never played with the ELK stack before and I'm going from the docs and blog articles I've found.

Any help is much appreciated.

Versions
Logstash: logstash/stable,now 1:7.17.1-1 amd64
Filebeat: filebeat/stable,now 7.17.1 amd64

Configurations

filebeat.yml

- type: syslog
  enabled: true
  format: auto
  protocol.udp:
    host: "1.1.1.1.148:9001" # not real ip
  tags: ["aruba-cxos", "aruba"]
  fields:
    log_type: "aruba-cxos"

- type: syslog
  enabled: true
  format: auto
  protocol.udp:
    host: "1.1.1.1:9002"
  tags: ["aruba-switchos", "aruba"]
  fields:
    log_type: "aruba-switchos"
#### skipping 
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["0.0.0.0:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

40-Elasticsearch-output.conf

input {
  beats {
    port => 5044
  }
}

output {
  if [@metadata][pipeline] {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        pipeline => "%{[@metadata][pipeline]}"
        }
  } else {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        }
  }
}

Netstat shows 9001 and 9002 are listening and when sending logs to 9002 I always see them coming in using tcpdump

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.