Cannot see parsed fields in kibana dashboard


(pramod choudhari) #1

This is my logstash.conf file:-
input {
beats {
port => 5044
}
}

filter {
if [log_type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
if ("_grokparsefailure" in [tags]) { drop {} }
}

output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

This is my Filebeat.yml file:-

filebeat:
prospectors:
-
paths:
- /var/log/auth.log
- /var/log/syslog
# - /var/log/*.log

  input_type: log

  fields:
     log_type: syslog

registry_file: /var/lib/filebeat/registry

output:
logstash:
hosts: ["107.109.101.20:5044"]
bulk_max_size: 1024

I cannot see syslog_timestamp, syslog_hostname , syslog_program, syslog_pid , syslog_message in the fields column in kibana.All I could see is the message as it is in the dashboard


I have gone through many topics discussed on your website but i'm stuck please help!


(Christian Dahlqvist) #2

If you look at the event in Kibana, you can see that this field is really named fields.log_type and not just log_type. You therefore need to change your conditional as follows:

if [fields][log_type] == "syslog" {

(pramod choudhari) #3

It worked thank you very much @Christian_Dahlqvist


(pramod choudhari) #4

Also @Christian_Dahlqvist can you please tell me what is the use of this section of code in logstash.conf file?

syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}

(Christian Dahlqvist) #5

It parses the syslog_timestamp field based on the specified patterns and populates the @timestamp field.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.