# Logstash filter user\_agent or geoip very slow

**URL:** <https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283>\
**Category:** Logstash\
**Created:** [March 2, 2016, 5:24pm UTC](https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283 "2016-03-02T17:24:42Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [March 2, 2016, 5:24pm UTC](https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283/1 "2016-03-02T17:24:42Z")

</div>

I am using logstash with RabbitMQ as input, elasticsearch as output, and user\_agent/geoip filter.

Configuration is pretty simple:

```
input
{
	rabbitmq
	{
        host => "queue"
        exchange => "bench"
        queue => "events"
        durable => true
        auto_delete => false
        threads => 1
    }
}

filter
{
	useragent
    {
        source => "ua"
        target => "ua"
        lru_cache_size => 5000
    }

    geoip
    {
        source => "ip"
        target => "geo"
        fields => ["country_name", "continent_code", "city_name", "location", "timezone", "real_region_name"]
    }
}

output
{
	elasticsearch
    {
        hosts => ["elasticsearch:9200"]
        index => "events-bench1"
        document_id => "%{id}"
        document_type => "events"
        manage_template => false
        flush_size => 2500
        workers => 4
    }
}

```

Without filters, logstash fetchs 2000 messages by second from RabbitMQ, pretty fast. But since I added filters, logstash consumes no more than 100 messages by second !! And I can see CPU at 100%.

I try to play with "pipeline-batch-size", "wokers" settings, but without success, what could be the best approach to scale logstash (horizontally is an option) so it can consume at least 500msgs/sec without burning my servers?

Here the RabbitMQ messages deliver rate:

[RabbitMQ messages deliver rate](http://b3.ms/PjL9yR7rAqXD)

---

<div class="post-metadata">

**Author:** ![arizonawayfarer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arizonawayfarer/32/44846_2.png) [@arizonawayfarer](https://discuss.elastic.co/u/arizonawayfarer)\
**Post date:** [March 2, 2016, 9:43pm UTC](https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283/2 "2016-03-02T21:43:01Z")

</div>

From what I understand, doing lookups in the geo database is pretty IO expensive. Your filtering is probably stuck waiting for IO operations on the disk. What sort of hardware are you running it on?

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [March 3, 2016, 6:54am UTC](https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283/3 "2016-03-03T06:54:08Z")

</div>

I am running it on my laptop MacOS i7, on Docker containers running on VirtualBox machine with 4 vCPU and 4 Gb RAM.

I believed geoip caches geo-database file! This is a good point, I will put logstash in a tmpfs folder and try again.

Thanks you for the idea.

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [March 3, 2016, 10:57am UTC](https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283/4 "2016-03-03T10:57:33Z")

</div>

It's a bit better indeed !

But I am afraid there is a mutex lock, and so, logstash pipeline is useless in this case.

I have better result with a single pipeline worker

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [March 3, 2016, 12:58pm UTC](https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283/5 "2016-03-03T12:58:59Z")

</div>

Made a PR:

> <https://github.com/logstash-plugins/logstash-filter-geoip/pull/63>

Which solves the mutex issue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:08am UTC](https://discuss.elastic.co/t/logstash-filter-user-agent-or-geoip-very-slow/43283/6 "2017-07-06T05:08:34Z")

</div>