The unknown output information in logstash

[antony.y (antony)]

I have used some output plugin in my logstash and conduct the different actions through the condition i've defined.

if "cisco" in [tags] {
    elasticsearch {
        hosts     => [ "ph71v-esn01.ae007.com:9200", "ph71v-esn02.ae007.com:9200" ]
        index    => "network-log-%{+YYYY.MM.dd}"
                    document_type => "Network Device"
                document_id   => "%{[@metadata][fingerprint]}"
    }
}

if "_grokparsefailure" in [tags] {
    file {
        path => "/var/log/logstash/fail-%{type}-%{+YYYY.MM.dd}.log"
    }
}
else {
    file {
        path => "/var/log/logstash/unknown_msg.log"
    }
}

i have some logs have been tagged by "cisco",but i can still find it in the unknow_msg.log,i do not know why, is there any logic mistake i've made ?

[magnusbaeck (Magnus Bäck)]

With your configuration all messages that don't have a _grokparsefailure tag will get routed to the unknown_msg.log file. In addition, if they have a cisco tag they will get routed to elasticsearch.

[antony.y (antony)]

the strange thing is that there should not be any unknown information ,you can see the input configuration
input {
tcp {
port => "8514"
type => "syslog-cisco"
add_field => {"Location" => "RCBC"}
}
udp {
port => "514"
type => "syslog-cisco"
add_field => {"Location" => "RCBC"}
}
}

filter {
if [type] == "syslog-cisco" {

# The switches are sending the same message to all syslog servers for redundancy, this allows us to
## only store the message in elasticsearch once by generating a hash of the message and using that as
## the document_id.
fingerprint {
  source              => [ "message" ]
  method              => "SHA1"
  key                 => "0123"
  concatenate_sources => true
  target              => "[@metadata][fingerprint]"
}

grok {

    # There are a couple of custom patterns associated with this filter.
	patterns_dir => [ "/opt/logstash/patterns" ]
	
	match => [
    "message", "<%{NUMBER:message_type_id}>%{NUMBER:internal_id}:%{SPACE}%{SYSLOGHOST:origin_hostname}:%{SPACE}%{CISCOTIMESTAMPTZ:cisco_timestamp}: \%%{WORD:facility}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:msg}",
	"message", "<%{NUMBER:message_type_id}>%{NUMBER:internal_id}:%{SPACE}%{SYSLOGHOST:origin_hostname}:%{SPACE}%{CISCOTIMESTAMP:cisco_timestamp}.*: \%%{WORD:facility}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:msg}",
	"message", "<%{NUMBER:message_type_id}>%{NUMBER:internal_id}:%{SPACE}%{SYSLOGHOST:origin_hostname}:%{SPACE}.*%{CISCOTIMESTAMP:cisco_timestamp}.*: \%%{WORD:facility}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:msg}",
    "message", "<%{NUMBER:message_type_id}>%{NUMBER:internal_id}:%{SPACE}%{SYSLOGHOST:origin_hostname}:.*\]:%{SPACE}%{CISCOTIMESTAMPTZ:cisco_timestamp}.*\%%{WORD:facility}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:msg}",
	"message", "<%{NUMBER:message_type_id}>%{NUMBER:internal_id}:%{SPACE}%{SYSLOGHOST:origin_hostname}:.*\]: %{NUMBER:sequencenumber}:%{SPACE}%{CISCOTIMESTAMPTZ:cisco_timestamp}.*\%%{WORD:facility}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:msg}",
	"message", "%{CISCOTIMESTAMPTZ:cisco_timestamp}%{SPACE}%{SYSLOGHOST:origin_hostname}%{SPACE}\%%{WORD:facility}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:msg}"
    ]
	add_tag => [ "cisco" ]
	remove_field => [ "cisco_timestamp" ]
	remove_field => [ "message_type_id" ]
	remove_field => [ "internal_id" ]
	remove_field => [ "sequencenumber" ]
}

if "cisco" in [tags] {

translate {
  field       => "facility"
  destination => "facility_full"

[magnusbaeck (Magnus Bäck)]

You're missing my point. With your configuration all messages that don't have a _grokparsefailure tag will get routed to the unknown_msg.log file even if they have a cisco tag. In addition, if they have a cisco tag they will get routed to elasticsearch.

I suspect the configuration you're looking for is this:

if "cisco" in [tags] {
    elasticsearch {
        hosts     => [ "ph71v-esn01.ae007.com:9200", "ph71v-esn02.ae007.com:9200" ]
        index    => "network-log-%{+YYYY.MM.dd}"
                    document_type => "Network Device"
                document_id   => "%{[@metadata][fingerprint]}"
    }
}
else if "_grokparsefailure" in [tags] {
    file {
        path => "/var/log/logstash/fail-%{type}-%{+YYYY.MM.dd}.log"
    }
}
else {
    file {
        path => "/var/log/logstash/unknown_msg.log"
    }
}```

[antony.y (antony)]

so it is! thank you so much ,it does work !