Syslog with logstash mapping

Elastic Stack Elasticsearch
1

Hi
I have logstash that get the syslog from other machines.
The conf.d file is like:

input {
  tcp {
    port => 514
    type => syslog
  }
  udp {
    port => 514
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { hosts => [":9200"] }
}

I want to use the feild message with terms in Visualize but there is no feild message to select.
I have read that this happen when the feild message is not having "keyword" so it can't be used to be aggregatable.
Is this the reall reason?

How can I add for the message feild:
"fields": {
"keyword": {
"type": "keyword"
}
}
Do I have to change the mapping? and how can I do so?

2

Hey,

you should not aggregate on the message field itself, because each field will be different (because almost every message already has a different timestamp). You should aggregate on the extract fields like host, program etc. This is also the reason why the message field does not support this and logstash configures the mapping the way it does.

Maybe you can talk about your use-case and why you want to aggregate so people can find another solution to your problem.

--Alex

3

Thanks
I want to create visualization like pie that will show top 5 hosts and number for events on each of this top 5 hosts
something like this:
logstash

4

then you dont need to aggregate on the message field but on the host field.

5

OK
Thanks

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.