# Useragent filter really slow

**URL:** <https://discuss.elastic.co/t/useragent-filter-really-slow/24759>\
**Category:** Logstash\
**Created:** [July 2, 2015, 12:45am UTC](https://discuss.elastic.co/t/useragent-filter-really-slow/24759 "2015-07-02T00:45:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Gabriel\_Rosca](https://avatars.discourse-cdn.com/v4/letter/g/e19adc/32.png) [@Gabriel\_Rosca](https://discuss.elastic.co/u/Gabriel_Rosca)\
**Post date:** [July 2, 2015, 12:45am UTC](https://discuss.elastic.co/t/useragent-filter-really-slow/24759/1 "2015-07-02T00:45:22Z")

</div>

Hello logstash community.

I have a problem with useragent filter. It is pailful slow. :(( Works but really slow. Same problem with the cidr filter.

If I use the filter I may get maybe 3 events per second if I am lucky.

I am running Logstash 1.5.1 on CentOS 6.6 with java-1.8.0-openjdk

VM: 32 VCPU with 32 workers for logstash and 8G RAM

MY setup: LSF ---\> REDIS ---\> LSI ----\> ES

If I don't use useragent filter ... the LSI it is really fast.

Here is my filter config:

filter {  
if [type] == "ironport" {  
grok {  
match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
}  
syslog\_pri { }  
date {  
match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]  
}  
mutate {  
replace =\> ["@message", "%{syslog\_message}"]  
}  
if [syslog\_program] == "SECLOG" {  
grok {  
match =\> {  
"syslog\_message" =\> "%{WORD:severity}: %{IP:user\_ip} (-|"%{WORD:domain}\%{NOTSPACE:user}@%{WORD:realm}") %{NOTSPACE} [%{HTTPDATE}] "%{WORD:request} (|%{URIPROTO:url\_proto}://)(?:%{URIHOST:url\_host})?(?:%{URIPATH:url\_path}(?:%{URIPARAM:url\_param})?)?" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NOTSPACE:result\_code}:%{NOTSPACE:code\_value} %{NUMBER} %{DATA:acl\_decision\_tag}-%{DATA:access\_policy}-%{DATA:identity}-%{DATA} \<%{GREEDYDATA:source\_csv}\> - %{NOTSPACE:url\_ip}, %{WORD:auth\_method}?, %{GREEDYDATA:user\_agent\_browser}" }  
}  
csv {  
source =\> "source\_csv"  
columns =\> ["category","reputation\_score","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove","app\_name","app\_type","remove","remove","Average\_bandwidth\_KB\_sec","remove","remove","remove","remove","remove","remove","remove","remove","remove","remove"]  
remove\_field =\> ["remove"]  
}  
if [syslog\_hostname] == "proxy01" or [syslog\_hostname] == "proxy02" {  
useragent {  
source =\> "user\_agent\_browser"  
}  
}  
mutate{  
remove\_field =\> ["source\_csv", "message", "host", "@version", "syslog\_message", "syslog\_program"]  
}  
if "\_grokparsefailure" in [tags] {  
drop { }  
}  
}  
}  
}

Here is a sample log:

Info: 172.16.92.32 - - [01/Jul/2015:20:41:07 -0400] "GET [http://live.lemde.fr/mux.json](http://live.lemde.fr/mux.json)" 304 0 TCP\_MISS:DIRECT 4 ALLOW\_WBRS\_12-DefaultGroup-A2TZ.noAuth.ID-NONE-NONE-NONE-DefaultGroup \<IW\_news,3.0,1,"-",-,-,-,-,"-",-,-,-,"-",-,-,"-","-",-,-,IW\_news,-,"-","-","Unknown","Unknown","-","-",986.00,0,-,"-","-",-,"-",-,-,"-","-"\> - 72.21.91.8, NONE, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36"

I get what I need but really really slow.

name Chrome  
os Windows 7  
os\_name Windows 7  
patch 2357

Any suggestions ?

Regards,  
Gabriel

---

<div class="post-metadata">

**Author:** ![Joshua\_Rich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshua_rich/32/44953_2.png) [@Joshua\_Rich](https://discuss.elastic.co/u/Joshua_Rich)\
**Post date:** [July 2, 2015, 1:59am UTC](https://discuss.elastic.co/t/useragent-filter-really-slow/24759/2 "2015-07-02T01:59:52Z")

</div>

Hi Gabriel,

The speed slowdown when using the useragent filter looks like a know issue, see [https://github.com/logstash-plugins/logstash-filter-useragent/issues/5](https://github.com/logstash-plugins/logstash-filter-useragent/issues/5). At this stage there is no workaround other than developing your won grok patterns to extract what you need out of the useragent part of the message. There may very well be good examples out on the net already.

---

<div class="post-metadata">

**Author:** ![Gabriel\_Rosca](https://avatars.discourse-cdn.com/v4/letter/g/e19adc/32.png) [@Gabriel\_Rosca](https://discuss.elastic.co/u/Gabriel_Rosca)\
**Post date:** [July 2, 2015, 1:46pm UTC](https://discuss.elastic.co/t/useragent-filter-really-slow/24759/3 "2015-07-02T13:46:20Z")

</div>

Thank you Joshua,

Was working just fine with logstash 1.4 not sure what happen.

Gabe

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:35am UTC](https://discuss.elastic.co/t/useragent-filter-really-slow/24759/4 "2017-07-06T05:35:44Z")

</div>