Problem with received_from not expanding

Elastic Stack Logstash
1

I just set up my first ELK cluster and I'm having a strange issue I can't track down.

I have logstash-forwarder running on the ELK host itself (set it up before I knew filebeat was a thing), and I have an external host running filebeat sending syslog messages to my ELK host. In my logstash config I have received_from set up in what looks like a fairly standard way:

filter {
    if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
        }
        syslog_pri { }
        date {
          match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
    }
}

Messages coming from my ELK host itself show up under its hostname in Kibana just fine. However, messages coming from my other host (using filebeat) show up with %{host} in the received_from field, like they're not getting expanded by (I presume) logstash. This is from logstash's stdout:

{
                 "message" => "Mar 23 14:43:19 juno postfix/anvil[2858]: statistics: max cache size 1 at Mar 23 14:33:29",
                "@version" => "1",
              "@timestamp" => "2016-03-23T18:43:19.000Z",
                    "beat" => {
        "hostname" => "juno",
            "name" => "juno"
    },
                   "count" => 1,
                  "fields" => nil,
              "input_type" => "log",
                  "offset" => 170393,
                  "source" => "/var/log/mail.log",
                    "type" => "syslog",
        "syslog_timestamp" => "Mar 23 14:43:19",
         "syslog_hostname" => "juno",
          "syslog_program" => "postfix/anvil",
              "syslog_pid" => "2858",
          "syslog_message" => "statistics: max cache size 1 at Mar 23 14:33:29",
             "received_at" => "2016-03-23T18:43:27.111Z",
           "received_from" => "%{host}",
    "syslog_severity_code" => 5,
    "syslog_facility_code" => 1,
         "syslog_facility" => "user-level",
         "syslog_severity" => "notice"
}

I'm using these versions:

  • elasticsearch 2.2.1
  • kibana 4.4.2
  • logstash 2.0.0
  • filebeat 1.1.2

All from elastic.co repositories/sources, on Debian stable.

2

Logstash doesn't add a host field to events received with the beats input. I'm not sure if there's a better way of doing that than having a ruby filter like this (untested):

filter {
  if not [host] {
    ruby {
      init => "require 'socket'"
      code => "event['host'] = Socket.gethostname"
    }
  }
}
3

Thanks, that is very helpful! I spent hours searching trying to figure out if I did anything wrong, but of course I went back just now and searched some more and came across this Github issue:

I also figured out I somehow installed logstash 2.0 instead of 2.2. Everything seems to be working now.

4

Sorry, I misread your question and thought you wanted the name of the receiving host. Yeah, if you want the origin host then you should definitely use the value that comes from beat.hostname or if that value is already put into host.