How to tell what fields can be examined/coded to in a Logstash filter?


(Russell Bateman) #1

If I want to write a filter that does something different depending on the provenance of a log entry, I am encouraged by examples to write

filter
{
  if [ source ] =~ "/var/log/acme/debug.log"
  {
    dissect
    {
      mapping =>
      {
        "message" => "..."
      }
    }
  }
  else # filter entries from other logfiles:
  {
    dissect
    {
      ...
    }
  }
}

However, it appears that source is not a field (though I see it in Kibana) available to me at this point. How can I know which fields Kibana displays (see below) are available to my filter logic? In Kibana, with no logic in my Logstash filter, I see:

filebeat-*
Selected fields
t  message
Available fields
O  @timestamp
t  @version
t  _id
t  _index
#  _score
t  _type
t  beat.hostname
t  beat.name
t  beat.version
t  host.name
t  input.type
#  offset
t  prospector.type
t  source
t  tags

Experimentation seems to confirm that I can see message, because that's what my dissect operates successfully on (when I put no conditionals into the filter). However, when I try to code different dissect statements based on a conditional, my filter fails to do anything--as if source didn't exist. In fact, my observation is that if I put any such conditional at all (to which I have no access?, i.e.: doesn't exist?), the whole filter fails--not even the else part of my filter works which works perfectly if that is the only code in my filter, and I don't even get the option of a "filebeat-*" index pattern in Kibana after.


(Russell Bateman) #2

The solution is in Elastic documentation
https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-log.html. The way to tell what fields are available in any given case is to:

  1. Fire log entries at Logstash with no defined filter.
  2. Create your index pattern in Kibana and examine the list of fields it associates with the data.
  3. Whatever is both in the list in #2 above and also in the Elastic documentation's list of exported fields can be coded to (i.e.: used in conditional expressions) in your filter.