# Logstash - newbie question - how to serach single event for a pattern, string **URL:** \ **Category:** Logstash\ **Created:** [May 23, 2016, 4:22pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742 "2016-05-23T16:22:35Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 23, 2016, 4:22pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/1 "2016-05-23T16:22:35Z")
Hi, I am just beginning my story with ELK, trying to understand grok and how to filter events Here is what I try to accomplish, I try to gather syslogs from routers and switches, here is example I 05/23/16 11:50:14 ports: port 14 is now off-line I 05/23/16 11:49:38 ports: port 14 is now on-line W 05/23/16 11:49:36 ports: port 14 PD Invalid Signature indication. I 05/23/16 11:49:35 ports: port 14 is Blocked by STP I 05/23/16 11:49:32 ports: port 14 is now off-line I 05/23/16 11:49:26 ip: VLAN101: network enabled on 10.101.0.130 I 05/23/16 11:49:26 ip: VLAN101: changing IP address to 10.101.0.130 All I want to do now is to catch string like "is now off-line" and then send emial alert, second part will be using Kibana to go through all logs easily, they can be raw. Log will be flowing from multiple devices, they will be different types so I just need to catch encountered string, mark it and execute (eg send email) input { tcp { type =\> "syslog" port =\> 514 } } input { udp { type =\> "syslog" port =\> 514 } } filter { if [type] == "syslog" { grok { check if the single sting( or strings from dictionary) exist in this message } if yes, mark it, tag it if not do nothing just send the raw event to elasticsearch} output { email { elasticsearch { } from =\> "myemail@somewhere.com" match =\> [ "problem1", "problem2","problem3" ] subject =\> "%{tagged\_event}" to =\> "you@example.com" via =\> "mail" body =\> "Alert something something: %{@message}" } } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 25, 2016, 5:35am UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/2 "2016-05-25T05:35:31Z")
This should work: ```auto filter { if "is now offline" in [message] { mutate { add_tag => ["port-offline"] } } ... } output { ... if "port-offline" in [tags] { email { ... } } } ``` You'll probably want to have a grok filter first to extract the timestamp in the `message` field into its own field. You might also want to use a regexp match instead of just using the `in` operator. > body =\> "Alert something something: %{@message}" Unless you're using Logstash 1.1 or something it's `message` and not `@message`. ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 25, 2016, 1:34pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/3 "2016-05-25T13:34:01Z")
I did small test and it does not seem to work, in case the syslogs comes in with "in now offline" in the message nothing happens which means that it does not work. Without that IF check I got event in the standard output input { udp { type =\> "syslog" port =\> 514 } } filter { if "is now offline" in [message] { mutate { add\_tag =\> ["port-offline"] } } } output { elasticsearch { } if "port-offline" in [tags] { stdout { } } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 25, 2016, 1:41pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/4 "2016-05-25T13:41:22Z")
What does such an event look like in Elasticsearch? ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 25, 2016, 4:00pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/5 "2016-05-25T16:00:00Z")
The point of my test is to see if the pattern is found then show that event on stdout { } I don't know how it looks in elastichsearch, I am just using it with kibana so all I see are log going through ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 25, 2016, 4:06pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/6 "2016-05-25T16:06:21Z")
> The point of my test is to see if the pattern is found then show that event on stdout { } Yes, but in order to debug why this doesn't work as expected I want to see what the resulting event looks like. > I don't know how it looks in elastichsearch, I am just using it with kibana so all I see are log going through All your events are sent to ES, including those that should've been sent to stdout. What do those events look like? Use Kibana's raw view (or whatever it's called) to see the event's JSON representation. ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 25, 2016, 4:22pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/7 "2016-05-25T16:22:04Z")
``` { "_index": "logstash-2016.05.25", "_type": "syslog", "_id": "AVToEqpIu0XuQF1O4IIc", "_score": null, "_source": { "message": "<14> May 25 09:20:17 10.0.0.6 ports: port 1 is now off-line", "@version": "1", "@timestamp": "2016-05-25T13:20:17.336Z", "type": "syslog", "host": "x.x.x.x" }, "fields": { "@timestamp": [ 1464182417336 ] }, "highlight": { "message": [ "<14> May 25 09:20:17 10.0.0.6 ports: port 1 is now @kibana-highlighted-field@off@/kibana-highlighted-field@-line" ] }, "sort": [ 1464182417336 ] } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 25, 2016, 4:40pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/8 "2016-05-25T16:40:10Z")
This: ``` if "is now offline" in [message] { ``` Needs to be ``` if "is now off-line" in [message] { ``` to match the actual message. ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 25, 2016, 4:43pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/9 "2016-05-25T16:43:15Z")
Thank you for your help ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 26, 2016, 1:14pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/10 "2016-05-26T13:14:36Z")
Going deeper with this, how can I compare array of strings or some kind of patterns from external file with eg set of IP. so instead of ``` if "is now off-line" in [message] { .... } ``` I would need to test array of 30 fixed patterns (stored as a string) form any kind of source against that message field If I got match next step will be do the same with set of 30 IPs and host field. How to load and compare sth like that? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 26, 2016, 1:25pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/11 "2016-05-26T13:25:56Z")
The translate filter should be able to help with that. ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 27, 2016, 6:38pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/12 "2016-05-27T18:38:27Z")
So I have some problem here understanding sth, let's say my file look looks like that: ``` "1.1.1.1": "WH" "test": "test_desc" "3": "abc" test: abcd ``` and config with translate filter like that: filter .... .... ``` if "1.1.1.1" in [host] { mutate { add_tag => ["WH", "%{space}"] } translate { field => "locationIP" destination => "space" dictionary_path => "/opt/logstash/bin/yaml/spaces_ip.yaml" add_tag => ["space"] fallback => "no match" refresh_interval => "60" } } ``` ..... What am I really comparing (entire file?) here against what ? Where is the argument that translate is using for comparison How to use that value later? Like to compare the match from first column against IP string and second column with name against sth else? I have noticed that there is no "space" tag in my output (elasticsearch -\> kibana) event though all look good ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 29, 2016, 12:41pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/13 "2016-05-29T12:41:32Z")
> What am I really comparing (entire file?) here against what ? Where is the argument that translate is using for comparison The translate filter looks up the contents of a field (the `field` option) against the lookup table and stores the result in another field (the `destination` option). It looks like you're getting this part. > How to use that value later? Like to compare the match from first column against IP string and second column with name against sth else? Not sure exactly what you mean here. > I have noticed that there is no "space" tag in my output (elasticsearch -\> kibana) event though all look good I don't know why your example didn't work. It would be easier to debug if you provided the actual event, preferably as presented by a `stdout { codec => rubydebug }` output. ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 31, 2016, 12:51pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/14 "2016-05-31T12:51:37Z")
Here is my confing ``` input { udp { type => "syslog" port => 514 } } if "1.1.1.1" in [host] { mutate { add_tag => ["WH"] add_tag => ["%{host}"] } translate { field => "location" destination => "space" dictionary_path => "/opt/logstash/bin/yaml/spaces_ip.yaml" add_tag => ["space2"] fallback => "no match" refresh_interval => "60" } } } output { elasticsearch { } stdout { codec => rubydebug } if "port-offline" in [tags] { email { ... } } } { "message" => "<14> May 31 08:43:02 10.0.0.6 mgr: SME TELNET from 10.0.0.1 - MANAGER Mode", "@version" => "1", "@timestamp" => "2016-05-31T12:43:03.259Z", "type" => "syslog", "host" => "1.1.1.1", "tags" => [ [0] "WH", [1] "1.1.1.1" ] } ``` Which idicates that translate filter was not used since I dont see extra "space2" tag added, why? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 31, 2016, 1:44pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/15 "2016-05-31T13:44:24Z")
Hmm. You're also not getting the fallback "no match" value. I don't know what's up here. ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 31, 2016, 4:29pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/16 "2016-05-31T16:29:44Z")
That is strange for me also, to make it even more strange I noticed that the very first time I run this confing, the translate filter worked properly because I saw extra tag added. No no matter what I do now, extra separate config file, different input files, dictionary, makes this translate filter not working at all. I even tried to install it again, no success. ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 31, 2016, 4:50pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/17 "2016-05-31T16:50:39Z")
Ok, got it, the reason was the "location" filed did not exist ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [May 31, 2016, 9:30pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/18 "2016-05-31T21:30:49Z")
Ok I am getting closer to what I need here is my filter ``` filter { mutate { add_field => { "location_IP" => "%{host}" } add_field => { "capture_msg" => "not much here yet" } } translate { field => "location_IP" destination => "site_name" dictionary_path => "/opt/logstash/bin/yaml/spaces_ip.yaml" add_tag => ["%{location_IP}"] fallback => "no match" refresh_interval => "60" } if [location_IP] == "no match" { mutate { add_tag => ["not matched"] } } else { #would like to see if any string (like "is off-line") from predefined set in the file matches any string in the %{message} that I already receive via syslog # my important_msg.yaml may look like below: # # "is now off-line": "is now off-line" # "TELNET": "TELNET" # #now my idea is to create possibly loop that copares array of values from that file against %{message} and try to find it within it translate { field => "capture_msg" # somehow assign my_array[0] .... [x] destination => "msg" dictionary_path => "/opt/logstash/bin/yaml/important_msg.yaml" add_tag => ["%{capture_msg}"] fallback => "not important" refresh_interval => "60" } } ``` How to approach this to have the easiest possible result here? Not sure if my approach is correct, If my idea is correct how to load those predefined strings into variable array type and compare in the loop against message? ---
**Author:** ![mke](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mke](https://discuss.elastic.co/u/mke)\ **Post date:** [June 5, 2016, 3:33pm UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/19 "2016-06-05T15:33:27Z")
I figured it out , just used ruby filter and made a loop There is one thing that bothers me, the official docs are so minimum, there is lack of multiple examples you could digest. It would be great to have more examples than add\_tag and filed under basically every section Anyway, thank you for help again ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:54am UTC](https://discuss.elastic.co/t/logstash-newbie-question-how-to-serach-single-event-for-a-pattern-string/50742/20 "2017-07-06T04:54:30Z")