How to conditionally set the type?

Elastic Stack Logstash
1

Hello

I am new to logstash (using logstash 2.0 beta) so please bear with me. I have log files in a JSON format similar to

{"TAGS":".source.s_src","SOURCEIP":"127.0.0.1","SEQNUM":"2","PROGRAM":"logstash","PRIORITY":"info","PID":"8509","MESSAGE":"Waiting logstash (pid 6757) to die...","LEGACY_MSGHDR":"logstash[8509]: ","HOST_FROM":"debian","HOST":"debian","FACILITY":"daemon","DATE":"Oct 23 16:10:32"}

I was planning to use a filter to set type (which then becomes _type in elasticsearch) depending on the value of PRIORITY:

filter {
  if [PRIORITY] in ["emergency", "crit"] {
    mutate { type => "syslog-important" }
  }
  else {
     mutate { type => "syslog-debug" }
  }
}

When running logstash I get the following message:

Error: The setting `type` in plugin `mutate` is obsolete and is no longer available. You can achieve this same behavior with the new conditionals, like: `if [type] == "sometype" { mutate { ... } }`. If you have any questions about this, you are invited to visit https://discuss.elastic.co/c/logstash and ask.

The docs are quite evasive:

Until the introduction of conditionals, the only way to selectively apply filters and outputs to some events was to set the type, tags and exclude_tags options. These configuration options have been deprecated for some time now and this release either removes them, or they are marked with the new obsolete tag. You will not be able to use them in your configuration file without a resulting error.

What should I do if I want to set type (or whatever is later understood by elasticsearch as being the document type) conditionally?

2

@WoJ there's a difference between the type directive in the plugin and a field called type. You are trying to set the latter with the former, and that won't work.

This should do what you're trying to do (if field "type" already exists):

mutate { 
  replace => { "type" => "THE_TYPE_YOU_WANT_SET" }
}

If there is not yet a "type":

mutate {
  add_field => { "type" => "THE_TYPE_YOU_WANT_SET" }
}
3

@theuntergeek: Thank you. As I understand it, the type directive , when used as part of file in input sets something which is later understood as the doc type in elasticsearch once the event is sent there (via the appropriate output). This is mapped to _type in ES. This is what works for me now:

input
{
  file
  {
    path => "/var/log/all_syslog_in_json.log"
    start_position => "beginning"
    codec => json
    sincedb_path => "/etc/logstash/db_for_watched_files.db"
    type => "syslog-ng"
  }
}

With this I get events in ES which have the document type set to "syslog-ng".

Now, instead of setting the type at the file level, I wanted to do it later, in a filter, based on some conditions.

From your answer I get that it is not possible to use the directive type anymore in a filter? To be frank I am not sure what type does in that context - there is some magic done at the output level which transforms it into _type when sending to ES.

I understand that I can have a field called type but it will just get added to teh other fields (like TAGS, SOURCEIP, ... in my example) - which not what I want.

What I would like to achieve is to modify whatever is set by the type directive in a filter so that it is then transformed (behind the scenes) into _type when using elasticsearch as the output plugin.

I tried

filter {
  if [PRIORITY] in ["emergency", "crit"] {
    type => "syslog-important"
  }
  else {
     type => "syslog-debug"
  }
}

but got the error

Error: Expected one of #, { at line 14, column 10 (byte 195) after filter {
  if [PRIORITY] in ["emergency", "crit"] {
    type

Sorry if this is obvious but I am trying to understand how this works.

4

You can't use type => any more in Logstash, except in the input block. Inside the input block, it sets the value of the type field, which sets the _type for Elasticsearch.

The use of type => in the filter and output blocks in previous versions of Logstash (and this use has been marked as deprecated for many versions) was to act as a conditional. Effectively, using type => "foo" inside a filter or output block would only accept events which were set to type => "foo". Once conditionals were added, this functionality was deprecated. It was, as you have discovered, a poor choice as its behavior was different between the input and filter+output blocks. Now that we're advancing to a new major version, the deprecated feature is being set to obsolete, to generate exactly the error message you saw. Unfortunately, the docs have not all been updated for 2.0 yet.

This is why you need to use mutate with replace and/or add_field now, as shown in my earlier example.

5

I understand thanks.

I set type following your example and I end up in ES with a field type (in _source) and a field _type in the meta data. They both have the same value, the one I used when setting typein logstash.

Should I deduce from this that the field called type in logstash is treated in a special way by the elasticsearch output plugin - by assuming that its value should also be used to populate _type before sending to ES ?

6

That is correct.