# How to filter data with Logstash before storing parsed data in Elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-filter-data-with-logstash-before-storing-parsed-data-in-elasticsearch/169071>\
**Category:** Logstash\
**Created:** [February 19, 2019, 4:50pm UTC](https://discuss.elastic.co/t/how-to-filter-data-with-logstash-before-storing-parsed-data-in-elasticsearch/169071 "2019-02-19T16:50:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Han\_Xu](https://avatars.discourse-cdn.com/v4/letter/h/f05b48/32.png) [@Han\_Xu](https://discuss.elastic.co/u/Han_Xu)\
**Post date:** [February 19, 2019, 4:50pm UTC](https://discuss.elastic.co/t/how-to-filter-data-with-logstash-before-storing-parsed-data-in-elasticsearch/169071/1 "2019-02-19T16:50:59Z")

</div>

I understand that Logstash is for aggregating and processing logs. I have NGIX logs and had Logstash config setup as:

```auto
filter {
 grok {
   match => ["message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
   overwrite => ["message"]
 }
 mutate {
   convert => ["response", "integer"]
   convert => ["bytes", "integer"]
   convert => ["responsetime", "float"]
 }
 geoip {
   source => "clientip"
   target => "geoip"
   add_tag => ["nginx-geoip"]
 }
 date {
   match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]
   remove_field => ["timestamp"]
 }
 useragent {
   source => "agent"
 }
}

output {
 elasticsearch {
   hosts => ["localhost:9200"]
   index => "weblogs-%{+YYYY.MM}"
   document_type => "nginx_logs"
 }
 stdout { codec => rubydebug }
}

```

This would parse the unstructured logs into a structured form of data, and store the data into monthly indexes.

What I discovered is that the majority of logs were contributed by robots/web-crawlers. In python I would filter them out by:

```auto
browser_names = browser_names[~browser_names.str.\
                              match('^[\w\W]*(google|bot|spider|crawl|headless)[\w\W]*, na=False)]

```

However, I would like to filter them out with Logstash so I can save a lot of disk space in Elasticsearch server. Is there a way to do that? Thanks in advance!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 19, 2019, 5:40pm UTC](https://discuss.elastic.co/t/how-to-filter-data-with-logstash-before-storing-parsed-data-in-elasticsearch/169071/2 "2019-02-19T17:40:02Z")

</div>

If there is a browser\_names field on the event then something like

```
if [browser_names] =~ /^[\w\W]*(google|bot|spider|crawl|headless)[\w\W]/ {
    drop {}
}
```

---

<div class="post-metadata">

**Author:** ![Han\_Xu](https://avatars.discourse-cdn.com/v4/letter/h/f05b48/32.png) [@Han\_Xu](https://discuss.elastic.co/u/Han_Xu)\
**Post date:** [February 19, 2019, 8:02pm UTC](https://discuss.elastic.co/t/how-to-filter-data-with-logstash-before-storing-parsed-data-in-elasticsearch/169071/3 "2019-02-19T20:02:58Z")

</div>

Thx so much! It works like a charm

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 19, 2019, 8:02pm UTC](https://discuss.elastic.co/t/how-to-filter-data-with-logstash-before-storing-parsed-data-in-elasticsearch/169071/4 "2019-03-19T20:02:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.