# Slow log file read from url for geoip **URL:** \ **Category:** Logstash\ **Created:** [December 2, 2015, 5:58am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126 "2015-12-02T05:58:12Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Hans](https://avatars.discourse-cdn.com/v4/letter/h/e19b73/32.png) [@Hans](https://discuss.elastic.co/u/Hans)\ **Post date:** [December 2, 2015, 5:58am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/1 "2015-12-02T05:58:12Z")
Hi All, I am currently facing challenges with geoip processing for url. Does anyone maybe have any suggestions how to increase the processing speed? Here is the filter: geoip { ``` source => "url" target => "geoip" database => "/usr/share/GeoIP/GeoLiteCity.dat" add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => ["[geoip][coordinates]", "float"] } ``` geoip { source =\> "url" } ``` geoip { source => "url" target => "geoIPASN" database => "/usr/share/GeoIP/GeoIP.dat" } ``` } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 2, 2015, 6:43am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/2 "2015-12-02T06:43:32Z")
No obvious rooms for improve AFAICT. What event rate are you getting? How do you know it's these filters that are slowing things down? ---
**Author:** ![Hans](https://avatars.discourse-cdn.com/v4/letter/h/e19b73/32.png) [@Hans](https://discuss.elastic.co/u/Hans)\ **Post date:** [December 2, 2015, 7:31am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/3 "2015-12-02T07:31:33Z")
When I take them out the file is pocessed by about 27K every few seconds. Is it possible that due to URL a DNS lookup is done for every record? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 2, 2015, 7:34am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/4 "2015-12-02T07:34:55Z")
Ah, right. Yes, if the source is a hostname then it'll take a DNS lookup to get an IP address to look up. Apart from making sure you have a fast caching DNS server you can increase the number of filter workers with the `-w` startup option. ---
**Author:** ![Hans](https://avatars.discourse-cdn.com/v4/letter/h/e19b73/32.png) [@Hans](https://discuss.elastic.co/u/Hans)\ **Post date:** [December 2, 2015, 8:26am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/5 "2015-12-02T08:26:45Z")
Magnus, I have been trying to search for a solution how to do a initial url to IP lookup and then only use the IP address field for the rest of the filters. Do you have any suggestions? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 2, 2015, 10:52am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/6 "2015-12-02T10:52:52Z")
Have you looked at the dns filter? ---
**Author:** ![Hans](https://avatars.discourse-cdn.com/v4/letter/h/e19b73/32.png) [@Hans](https://discuss.elastic.co/u/Hans)\ **Post date:** [December 2, 2015, 5:50pm UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/7 "2015-12-02T17:50:35Z")
With the GeoIP information I am not getting any information in the geoip.location field that seems to be populated by default. I have added the following as the field that contains the information is called url geoip { source =\> "url" target =\> "geoip" database =\> "/etc/logstash/GeoLiteCity.dat" } The output is still ![](https://us1.discourse-cdn.com/elastic/original/2X/f/f222abda3ee00e8094ae89f25084954f32444f71.png) any suggestions why this field is not populated? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 3, 2015, 6:57am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/8 "2015-12-03T06:57:57Z")
It looks like you're trying to populate `geoip.location` with the contents of the `LATITUDE` and `LONGITUDE` fields but there are no such fields. If you show us your configuration we can help further. ---
**Author:** ![Hans](https://avatars.discourse-cdn.com/v4/letter/h/e19b73/32.png) [@Hans](https://discuss.elastic.co/u/Hans)\ **Post date:** [December 3, 2015, 11:20am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/9 "2015-12-03T11:20:39Z")
Here is the full configuration: input { file { type =\> "BIND\_DNS" path =\> ["/data/bind"] start\_position =\> "beginning" } } filter { grok { match =\> ["message","(?%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME}) queries: info: client %{IPORHOST:clientip}#%{NUMBER:port}: query: (?[a-z0-9-]+.[a-z0-9-]+\S+) IN %{WORD:recType} + (%{IPORHOST:DNSIP})"] } dns { add\_field =\> ["URL", "FQDN"] } dns { resolve =\> ["URL"] action =\> ["replace"] } geoip { source =\> "url" target =\> "geoip" database =\> "/usr/share/GeoIP/GeoLiteCity.dat" # add\_field =\> ["[geoip][coordinates]", "%{[geoip][longitude]}" ] # add\_field =\> ["[geoip][coordinates]", "%{[geoip][latitude]}" ] ``` } ``` # mutate { # convert =\> ["[geoip][coordinates]", "float"] # } geoip { source =\> "geoip.ip" } geoip { source =\> "geoip.ip" target =\> "geoIPASN" database =\> "/usr/share/GeoIP/GeoIP.dat" } } output { elasticsearch { protocol =\> "node" host =\> "localhost" cluster =\> "elasticsearch" } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 3, 2015, 11:42am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/10 "2015-12-03T11:42:58Z")
Is this really the only configuration you have? No other files in /etc/logstash/conf.d that you're forgetting about? I'm asking because I'm pretty sure none of the standard plugins attempt to reference any `LATITUDE` or `LONGITUDE` fields via the %{fieldname} notation. ---
**Author:** ![Hans](https://avatars.discourse-cdn.com/v4/letter/h/e19b73/32.png) [@Hans](https://discuss.elastic.co/u/Hans)\ **Post date:** [December 3, 2015, 6:10pm UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/11 "2015-12-03T18:10:03Z")
No only one single file in that directory with the configuration provided? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:20am UTC](https://discuss.elastic.co/t/slow-log-file-read-from-url-for-geoip/36126/12 "2017-07-06T05:20:03Z")