Logstash - newbie question - how to serach single event for a pattern, string

Elastic Stack Logstash
#1

Hi, I am just beginning my story with ELK, trying to understand grok and how to filter events

Here is what I try to accomplish, I try to gather syslogs from routers and switches, here is example

I 05/23/16 11:50:14 ports: port 14 is now off-line
I 05/23/16 11:49:38 ports: port 14 is now on-line
W 05/23/16 11:49:36 ports: port 14 PD Invalid Signature indication.
I 05/23/16 11:49:35 ports: port 14 is Blocked by STP
I 05/23/16 11:49:32 ports: port 14 is now off-line
I 05/23/16 11:49:26 ip: VLAN101: network enabled on 10.101.0.130
I 05/23/16 11:49:26 ip: VLAN101: changing IP address to 10.101.0.130

All I want to do now is to catch string like "is now off-line" and then send emial alert, second part will be using Kibana to go through all logs easily, they can be raw.

Log will be flowing from multiple devices, they will be different types so I just need to catch encountered string, mark it and execute (eg send email)

input {
tcp {
type => "syslog"
port => 514
}
}
input {
udp {
type => "syslog"
port => 514
}
}

filter {
if [type] == "syslog" {

grok { check if the single sting( or strings from dictionary) exist in this message }
if yes, mark it, tag it if not do nothing just send the raw event to elasticsearch}

output {

email {
elasticsearch {
}
from => "myemail@somewhere.com"
match => [
"problem1", "problem2","problem3"
]
subject => "%{tagged_event}"
to => "you@example.com"
via => "mail"
body => "Alert something something: %{@message}"
}
}

}

#2

This should work:

filter {
  if "is now offline" in [message] {
    mutate {
      add_tag => ["port-offline"]
    }
  }
  ...
}

output {
  ...
  if "port-offline" in [tags] {
    email {
      ...
    }
  }
}

You'll probably want to have a grok filter first to extract the timestamp in the message field into its own field. You might also want to use a regexp match instead of just using the in operator.

body => "Alert something something: %{@message}"

Unless you're using Logstash 1.1 or something it's message and not @message.

2 Likes
#3

I did small test and it does not seem to work, in case the syslogs comes in with "in now offline" in the message nothing happens which means that it does not work. Without that IF check I got event in the standard output

input {
udp {
type => "syslog"
port => 514
}
}

filter {
if "is now offline" in [message] {
mutate {
add_tag => ["port-offline"]
}
}
}

output {
elasticsearch {
}
if "port-offline" in [tags] {
stdout { }
}

}

#4

What does such an event look like in Elasticsearch?

#5

The point of my test is to see if the pattern is found then show that event on stdout { }

I don't know how it looks in elastichsearch, I am just using it with kibana so all I see are log going through

#6

The point of my test is to see if the pattern is found then show that event on stdout { }

Yes, but in order to debug why this doesn't work as expected I want to see what the resulting event looks like.

I don't know how it looks in elastichsearch, I am just using it with kibana so all I see are log going through

All your events are sent to ES, including those that should've been sent to stdout. What do those events look like? Use Kibana's raw view (or whatever it's called) to see the event's JSON representation.

#7 
{
  "_index": "logstash-2016.05.25",
  "_type": "syslog",
  "_id": "AVToEqpIu0XuQF1O4IIc",
  "_score": null,
  "_source": {
    "message": "<14> May 25 09:20:17 10.0.0.6 ports:  port 1 is now off-line",
    "@version": "1",
    "@timestamp": "2016-05-25T13:20:17.336Z",
    "type": "syslog",
    "host": "x.x.x.x"  
  },
  "fields": {
    "@timestamp": [
      1464182417336
    ]
  },
  "highlight": {
    "message": [
      "<14> May 25 09:20:17 10.0.0.6 ports:  port 1 is now @kibana-highlighted-field@off@/kibana-highlighted-field@-line"
    ]
  },
  "sort": [
    1464182417336
  ]
}
#8

This:

if "is now offline" in [message] {

Needs to be

if "is now off-line" in [message] {

to match the actual message.

#9

Thank you for your help

#10

Going deeper with this, how can I compare array of strings or some kind of patterns from external file with eg set of IP.

so instead of

if "is now off-line" in [message] {
  ....
}

I would need to test array of 30 fixed patterns (stored as a string) form any kind of source against that message field
If I got match next step will be do the same with set of 30 IPs and host field.

How to load and compare sth like that?

#11

The translate filter should be able to help with that.

#12

So I have some problem here understanding sth, let's say

my file look looks like that:

"1.1.1.1": "WH"
"test": "test_desc"
"3": "abc"
test: abcd

and config with translate filter like that:

filter ....
....

  if "1.1.1.1" in [host] {
    mutate {
      add_tag => ["WH", "%{space}"]
    }

  translate {
        field => "locationIP"
        destination => "space"
        dictionary_path => "/opt/logstash/bin/yaml/spaces_ip.yaml"
        add_tag => ["space"]
        fallback => "no match"
        refresh_interval => "60"
   } }

.....

What am I really comparing (entire file?) here against what ? Where is the argument that translate is using for comparison
How to use that value later? Like to compare the match from first column against IP string and second column with name against sth else?
I have noticed that there is no "space" tag in my output (elasticsearch -> kibana) event though all look good

#13

What am I really comparing (entire file?) here against what ? Where is the argument that translate is using for comparison

The translate filter looks up the contents of a field (the field option) against the lookup table and stores the result in another field (the destination option). It looks like you're getting this part.

How to use that value later? Like to compare the match from first column against IP string and second column with name against sth else?

Not sure exactly what you mean here.

I have noticed that there is no "space" tag in my output (elasticsearch -> kibana) event though all look good

I don't know why your example didn't work. It would be easier to debug if you provided the actual event, preferably as presented by a stdout { codec => rubydebug } output.

#14

Here is my confing

input {
 udp {
    type => "syslog"
    port => 514
  }
}
    
  if "1.1.1.1" in [host] {
    mutate {
      add_tag => ["WH"]
      add_tag => ["%{host}"]
    }
    
  translate {
        field => "location"
        destination => "space"
        dictionary_path => "/opt/logstash/bin/yaml/spaces_ip.yaml"
        add_tag => ["space2"]
fallback => "no match"
refresh_interval => "60"
      }
   }
}
output {
  elasticsearch {
  }
  stdout { codec => rubydebug }
  if "port-offline" in [tags] {
    email {
      ...
    }
}
} 



{
       "message" => "<14> May 31 08:43:02 10.0.0.6 mgr:  SME TELNET from 10.0.0.1 - MANAGER Mode",
      "@version" => "1",
    "@timestamp" => "2016-05-31T12:43:03.259Z",
          "type" => "syslog",
          "host" => "1.1.1.1",
          "tags" => [
        [0] "WH",
        [1] "1.1.1.1"
    ]
}

Which idicates that translate filter was not used since I dont see extra "space2" tag added, why?

#15

Hmm. You're also not getting the fallback "no match" value. I don't know what's up here.

#16

That is strange for me also, to make it even more strange I noticed that the very first time I run this confing, the translate filter worked properly because I saw extra tag added. No no matter what I do now, extra separate config file, different input files, dictionary, makes this translate filter not working at all. I even tried to install it again, no success.

#17

Ok, got it, the reason was the "location" filed did not exist

#18

Ok I am getting closer to what I need here is my filter

filter {
  mutate {
    add_field => { "location_IP" => "%{host}" }
    add_field => { "capture_msg" => "not much here yet" }
  }
  translate {
        field => "location_IP"
        destination => "site_name"
        dictionary_path => "/opt/logstash/bin/yaml/spaces_ip.yaml"
        add_tag => ["%{location_IP}"]
        fallback => "no match"
        refresh_interval => "60"
  }
  if [location_IP] == "no match" {
      mutate {
        add_tag => ["not matched"]
    }
  } else {
 #would like to see if any string (like "is off-line") from predefined set in the file matches any string in the %{message} that I already receive via syslog
 # my important_msg.yaml may look like below:
 #
 # "is now off-line": "is now off-line"
 # "TELNET": "TELNET"
 #
  #now my idea is to create possibly loop that copares array of values from that file against %{message}  and try to find it within it
        translate {
        field => "capture_msg"  # somehow assign my_array[0]  .... [x]
        destination => "msg"
        dictionary_path => "/opt/logstash/bin/yaml/important_msg.yaml"
        add_tag => ["%{capture_msg}"]
        fallback => "not important"
        refresh_interval => "60"
       }
  }

How to approach this to have the easiest possible result here?
Not sure if my approach is correct, If my idea is correct how to load those predefined strings into variable array type and compare in the loop against message?

#19

I figured it out , just used ruby filter and made a loop

There is one thing that bothers me, the official docs are so minimum, there is lack of multiple examples you could digest. It would be great to have more examples than add_tag and filed under basically every section

Anyway, thank you for help again

#20