# How to use if condition to filter spider event gracefully in logstash

**URL:** <https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [May 21, 2021, 4:58am UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606 "2021-05-21T04:58:31Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![stille](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stille/32/89061_2.png) [@stille](https://discuss.elastic.co/u/stille)\
**Post date:** [May 21, 2021, 4:58am UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606/1 "2021-05-21T04:58:31Z")

</div>

I can use drop event in filebeat to filter spider logs , just like following:

```
  - drop_event:
      when:
        or:
          - contains:
              message: FacebookBot
          - contains:
              message: Googlebot
            ...

```

How to do it in logstash ? There are so many spider event , how can i use if or condition just like following:

```
if ["TwitterBot","FacebookBot","Googlebot","AppleBot","xxx","xxxx"] in [message] {
    drop {}
}

```

It's desn't work..

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 21, 2021, 2:20pm UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606/2 "2021-05-21T14:20:02Z")

</div>

See [here](https://discuss.elastic.co/t/how-do-you-do-multiple-or-s-in-a-filter/208922/2).

---

<div class="post-metadata">

**Author:** ![stille](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stille/32/89061_2.png) [@stille](https://discuss.elastic.co/u/stille)\
**Post date:** [May 22, 2021, 3:05am UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606/3 "2021-05-22T03:05:54Z")

</div>

Thanks , it's working !

---

<div class="post-metadata">

**Author:** ![cknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cknz/32/9640_2.png) [@cknz](https://discuss.elastic.co/u/cknz)\
**Post date:** [May 22, 2021, 11:34am UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606/4 "2021-05-22T11:34:06Z")

</div>

For a more complete solution around identifying bots, try using the '[useragent](https://www.elastic.co/guide/en/logstash/current/plugins-filters-useragent.html)' filter. One of the fields that will add to your events is 'device', which typically shows various phone models, 'Other' for typical web-browsers, and 'Spider'.

I've used it in production for many years; its one of the most useful plugins for triaging web-server issues.

---

<div class="post-metadata">

**Author:** ![stille](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stille/32/89061_2.png) [@stille](https://discuss.elastic.co/u/stille)\
**Post date:** [May 22, 2021, 12:57pm UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606/5 "2021-05-22T12:57:49Z")

</div>

Thanks. this is the perfect solution right now.  
I had using useragent filter already, but I did not notice the `client.agent.device`, it's already match the spider automatic for me.  
Thanks again.

---

<div class="post-metadata">

**Author:** ![stille](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stille/32/89061_2.png) [@stille](https://discuss.elastic.co/u/stille)\
**Post date:** [May 22, 2021, 1:52pm UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606/6 "2021-05-22T13:52:41Z")

</div>

But i'm trying like this:

```
if [client.agent] != "-" {
    useragent {
        source => "useragent"
        target => "client.agent"
    }
}

```

Now i can see "iOS" "Other" "Spider" in "client.agent.device" fields.  
When i add a filter like following , it doesn't work

```
if [client][agent][device] =~ "Spider" {
    drop {}
}
```

---

<div class="post-metadata">

**Author:** ![cknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cknz/32/9640_2.png) [@cknz](https://discuss.elastic.co/u/cknz)\
**Post date:** [May 22, 2021, 11:39pm UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606/7 "2021-05-22T23:39:29Z")

</div>

What does an entire record look like? Can you share the entire filter{} so we can see these pieces in context?

PS. I would suggest its most useful to keep the spider activity and use a filter in Kibana etc. Gives much greater visibility with regard to what effect spiders are having (particularly for correlating outages and performance analysis. It would also give you information for making informed decisions around rate-limiting based on the likes of user-agents.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 19, 2021, 11:40pm UTC](https://discuss.elastic.co/t/how-to-use-if-condition-to-filter-spider-event-gracefully-in-logstash/273606/8 "2021-06-19T23:40:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.