Cannot see parsed fields in kibana dashboard

This is my logstash.conf file:-
input {
beats {
port => 5044
}
}

filter {
if [log_type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
if ("_grokparsefailure" in [tags]) { drop {} }
}

output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

This is my Filebeat.yml file:-

filebeat:
prospectors:
-
paths:
- /var/log/auth.log
- /var/log/syslog
# - /var/log/*.log

  input_type: log

  fields:
     log_type: syslog

registry_file: /var/lib/filebeat/registry

output:
logstash:
hosts: ["107.109.101.20:5044"]
bulk_max_size: 1024

I cannot see syslog_timestamp, syslog_hostname , syslog_program, syslog_pid , syslog_message in the fields column in kibana.All I could see is the message as it is in the dashboard


I have gone through many topics discussed on your website but i'm stuck please help!

If you look at the event in Kibana, you can see that this field is really named fields.log_type and not just log_type. You therefore need to change your conditional as follows:

if [fields][log_type] == "syslog" {

It worked thank you very much @Christian_Dahlqvist

Also @Christian_Dahlqvist can you please tell me what is the use of this section of code in logstash.conf file?

syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}

It parses the syslog_timestamp field based on the specified patterns and populates the @timestamp field.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.