# Selectively filtering messages log

**URL:** <https://discuss.elastic.co/t/selectively-filtering-messages-log/55012>\
**Category:** Logstash\
**Created:** [July 8, 2016, 3:38am UTC](https://discuss.elastic.co/t/selectively-filtering-messages-log/55012 "2016-07-08T03:38:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kennedy\_Kan1](https://avatars.discourse-cdn.com/v4/letter/k/cc9497/32.png) [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\
**Post date:** [July 8, 2016, 3:38am UTC](https://discuss.elastic.co/t/selectively-filtering-messages-log/55012/1 "2016-07-08T03:38:19Z")

</div>

I have come across with some issues in logging some data.

This log file contains various data.

```
2016-07-07 13:30:02 [Main] ***Program start***
2016-07-07 13:30:02 [UnzipFile] Before file collection
2016-07-07 13:30:02 [GetZipCol] Start get sorted zip file collection
2016-07-07 13:30:02 [GetZipCol] End get sorted zip file collection
2016-07-07 13:30:02 [Main] [ERROR] No unzip file
2016-07-07 13:30:03 [Main] ***Program end***

```

This is my output part in conf

```
if [Message] == " ***Program start***" {
	elasticsearch { 
	hosts => ["localhost:9200"] 
	index => "log-%{+YYYY.MM.dd}"
	template => "C:/logstash/log.json"
	template_overwrite => true
	}	
}
if [Message] == " ***Program end***" {
	elasticsearch { 
	hosts => ["localhost:9200"] 
	index => "log-%{+YYYY.MM.dd}"
	template => "C:/logstash/log.json"
	template_overwrite => true
	}	
}  
if [Level] =~ /.+/ {
	elasticsearch { 
	hosts => ["localhost:9200"] 
	index => "log-%{+YYYY.MM.dd}"
	template => "C:/logstash/log.json"
	template_overwrite => true
	}	
}

```

If I **only want to grasp the event when the Program starts and ends and also the events with errors** while the other events can be dropped. **However** , according to what I have written. I **can only grasp the data with [Error]**. How should I also grasp the other data? And will there be a simpler way of doing that instead of typing 3 if conditional statements?  
Thanks.

---

<div class="post-metadata">

**Author:** ![Kennedy\_Kan1](https://avatars.discourse-cdn.com/v4/letter/k/cc9497/32.png) [@Kennedy\_Kan1](https://discuss.elastic.co/u/Kennedy_Kan1)\
**Post date:** [July 8, 2016, 4:52am UTC](https://discuss.elastic.co/t/selectively-filtering-messages-log/55012/2 "2016-07-08T04:52:46Z")

</div>

You can do it in this way. In filter part of conf, use add tag to group the records you need

```
if [Progress] in [" ***Program start***", " ***Program end***"] or [Level] =~ /.+/ {
    	mutate {
    		add_tag => "send_to_es"
    	}
    }

```

Then in the output part, add this.

```
	if "send_to_es" in [tags] {
  		elasticsearch { 
		hosts => ["localhost:9200"] 
		index => "log-%{+YYYY.MM.dd}"
		template => "C:/logstash/log.json"
		template_overwrite => true
		}	
	}

```

This will only add those with the tags into ES. One thing yo should not miss is that you should add `\r` at the end of your matching message in your grok part.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:49am UTC](https://discuss.elastic.co/t/selectively-filtering-messages-log/55012/3 "2017-07-06T04:49:00Z")

</div>