Filebeat logs reaching Logstash but failing to create visible indices in Kibana

Elastic Stack Logstash
1
  • Logstash successfully receives logs from Filebeat.
  • However, no index is visible in Kibana when using the custom index + pipeline block.
  • If I remove this block:
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    pipeline => "%{[@metadata][pipeline]}"
    then the logs appear, but only inside the hidden system index:
    .ds-logs-generic-default-2025.09.09-000001

Expected Behavior : I want the logs to be stored in a visible, custom index (e.g., filebeat-8.15.0-2025.09.09) rather than only going into the hidden .ds-logs-generic-default-* data stream.

Notes :

  • Filebeat is installed in Machine 1 (192.168.1.199)
  • Logstash, Elasticsearch and Kibana are installed in Machine 2 (192.168.1.205)

Environment Setup :
Machine 1 (Filebeat, 192.168.1.199)

/etc/filebeat/modules.d/system.yml

  • module: system
    syslog:
    enabled: true
    var.paths: ["/var/log/syslog"]
    auth:
    enabled: true
    var.paths: ["/var/log/auth.log"]

/etc/filebeat/filebeat.yml

output.logstash:
hosts: ["192.168.1.205:5044"]
ssl.certificate_authorities: ["/etc/filebeat/certs/http_ca.crt"]
ssl.certificate: "/etc/filebeat/certs/filebeat.crt"
ssl.key: "/etc/filebeat/certs/filebeat.key"

Machine 2 (Logstash, 192.168.1.205)

/etc/logstash/conf.d/task.conf

input {
beats {
port => 5044
ssl_enabled => true
ssl_certificate => "/etc/logstash/certs/logstash.crt"
ssl_key => "/etc/logstash/certs/logstash.key"
}
}

output {
elasticsearch {
hosts => ["https://192.168.1.205:9200"]
ssl_certificate_authorities => "/etc/logstash/certs/http_ca.crt"
user => "elastic"
password => "cD-_sJHg3RJehMH7IRwT"
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
pipeline => "%{[@metadata][pipeline]}"
}
stdout { codec => rubydebug }
}