Filter problems


(Jack ELK West) #1

Hey guys I'm trying to match a string in a filter. But the output isn't giving me what i'm expecting.

The input is:
<36>Nov 02 15:48:57 LCE: [not-matched] 0.0.0.0:0 -> 10.1.116.173:0 ::

filter{
grok{
	match => {"message" => "<%{BASE10NUM:LCE_log_num}>%{SYSLOGTIMESTAMP:LCE_time} %{PROG:header_type}: \[matched] %{IP:Source_IP}:%{BASE10NUM:Source_Port} -> %{IP:Destination_IP}:%{BASE10NUM:Destination_Port} ::%{GREEDYDATA:Message_Data}" }
	add_field => { "sort_num" => "%{LCE_log_num}" }
    }
if "_grokparsefailure" in [tags] {
    grok {
	remove_tag => [ "_grokparsefailure" ]
	add_tag => [ "unmatched" ]
	break_on_match => true
    }
  }	
}
ouput {	
if "unmatched" in [tags] {
	file {
  	path => "C:\ELK\running\logstash-2.0.0\test\JackTest\unmatched.txt"
    }
elasticsearch {
}

What I want is for every log that fails to match to be output to a file and not elastic. What is happening for me is i'm getting the grokparsefailure and its not matching my if statement stripping that and adding unmatched.

Any help is greatly appreciated

Thanks


(Magnus Bäck) #2

That's because your second grok filter doesn't have any expressions to match so it doesn't count as successful, and because of that remove_tag and add_tag won't do anything. But you're overcomplicating things. Just set tag_on_failure for the first grok and drop the second one.


(Jack ELK West) #3

Thanks I was looking for something like that I must have looked over it. You the man


(Jack ELK West) #4

How can I have these events not go into elasticsearch though and just into a file?


(Magnus Bäck) #5

Just wrap the outputs in a conditional, similar to what you had in the filter section of your first message in this thread.


(Jack ELK West) #6

I just tried this and It didn't fix it for me. Do you mind writing out how it should look? (the output is still writing to elastic)


(Magnus Bäck) #7
output {
  if "unmatched" in [tags] {
    file {
      ...
    }
  } else {
    elasticsearch {
      ...
    }
  }
}

(Jack ELK West) #8

That is exactly what I had. I just changed it and wrote:

    if "multiline" in [tags] or "_xmlparsefailure" in [tags] or "_grokparsefailure" in [tags] or "unmatched" not in [tags] {
       file{
    }
}

(system) #9