Logstash 6.x & type

#1

Hello.
I'm tired of the warnings in my Logstash log file and would like to fix it:

[2018-12-06T16:34:17,200][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}

And below is an extract of my Logstash input/output files:

Input file:
udp {
port => 5516
type => "apc-log"
}
udp {
port => 5517
type => "ltm-log"
}
udp {
port => 5518
type => "firewall-traffic-log"
}
udp {
port => 5519
type => "firewall-threat-log"
}

Output file:

} else if [type] == "apc-log" {
  elasticsearch {
    hosts => ["els04","els03"]
    user => elastic
    password => elastic
    #sniffing => true
    manage_template => false
    index => "apc-%{+YYYY.MM.dd}"
  }
} else if [type] == "ltm-log" {
  elasticsearch {
    hosts => ["els04","els03"]
    user => elastic
    password => elastic
    #sniffing => true
    manage_template => false
    index => "ltm-%{+YYYY.MM.dd}"
  }
} else if [type] == "firewall-traffic-log" {
  elasticsearch {
    hosts => ["els04","els03"]
    user => elastic
    password => elastic
    #sniffing => true
    manage_template => false
    index => "firewall-%{+YYYY.MM.dd}"
   }
} else if [type] == "firewall-threat-log" {
  elasticsearch {
    hosts => ["els04","els03"]
    user => elastic
    password => elastic
    #sniffing => true
    manage_template => false
    index => "firewall-%{+YYYY.MM.dd}"
}

What other options can I use to seperate out inputs to different pipelines?

Thanks.

(Aaron Daisley) #2

Logstash 6.x introduced the Multiple Pipelines feature that allows you to create several config files (each containing an input, filter and output) and this eliminates the need for the type field.

#3

Hi,

The types in indices are deprecated since version 6 of the stack, and will be remove in the v7. See this article for further details.

In short, you should remove the lines type => "apc-log" and add a field instead (or a metadata field).
Then in your output you just have to check the value of this field to select the output.

#4

Hi and thanks.
I have over 18 'if else' statements in my input and output files.
I would like to keep them as they are if at all possible (not have to break each 'if else' into seperate files.
Any pointers on how to best replace the 'types' with tags? I did try this but no go...

} else if [tag] == "rsyslog-log" {
  elasticsearch {
    hosts => ["els03","els04"]
    user => elastic
    password => elastic
    #sniffing => true
    manage_template => false
    index => "rsyslog-%{+YYYY.MM.dd}"
    #document_type => "rsyslog-log"
  }
(Aaron Daisley) #5

Try

if ("foo" in [tags]) (tags not tag)

instead and see if that works? Obviously it goes without saying but make sure you've assigned the type correctly and use the stdout{} output plugin to double-check the tag is being assigned correctly.