Filter problems

Elastic Stack Logstash
1

Hey guys I'm trying to match a string in a filter. But the output isn't giving me what i'm expecting.

The input is:
<36>Nov 02 15:48:57 LCE: [not-matched] 0.0.0.0:0 -> 10.1.116.173:0 ::

filter{
grok{
	match => {"message" => "<%{BASE10NUM:LCE_log_num}>%{SYSLOGTIMESTAMP:LCE_time} %{PROG:header_type}: \[matched] %{IP:Source_IP}:%{BASE10NUM:Source_Port} -> %{IP:Destination_IP}:%{BASE10NUM:Destination_Port} ::%{GREEDYDATA:Message_Data}" }
	add_field => { "sort_num" => "%{LCE_log_num}" }
    }
if "_grokparsefailure" in [tags] {
    grok {
	remove_tag => [ "_grokparsefailure" ]
	add_tag => [ "unmatched" ]
	break_on_match => true
    }
  }	
}
ouput {	
if "unmatched" in [tags] {
	file {
  	path => "C:\ELK\running\logstash-2.0.0\test\JackTest\unmatched.txt"
    }
elasticsearch {
}

What I want is for every log that fails to match to be output to a file and not elastic. What is happening for me is i'm getting the grokparsefailure and its not matching my if statement stripping that and adding unmatched.

Any help is greatly appreciated

Thanks

2

That's because your second grok filter doesn't have any expressions to match so it doesn't count as successful, and because of that remove_tag and add_tag won't do anything. But you're overcomplicating things. Just set tag_on_failure for the first grok and drop the second one.

1 Like
3

Thanks I was looking for something like that I must have looked over it. You the man

4

How can I have these events not go into elasticsearch though and just into a file?

5

Just wrap the outputs in a conditional, similar to what you had in the filter section of your first message in this thread.

1 Like
6

I just tried this and It didn't fix it for me. Do you mind writing out how it should look? (the output is still writing to elastic)

7 
output {
  if "unmatched" in [tags] {
    file {
      ...
    }
  } else {
    elasticsearch {
      ...
    }
  }
}
8

That is exactly what I had. I just changed it and wrote:

    if "multiline" in [tags] or "_xmlparsefailure" in [tags] or "_grokparsefailure" in [tags] or "unmatched" not in [tags] {
       file{
    }
}