# Logstash Geolocate IP Address into Elasticsearch

**URL:** <https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246>\
**Category:** Logstash\
**Created:** [February 15, 2017, 7:38pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246 "2017-02-15T19:38:05Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bondo](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@Bondo](https://discuss.elastic.co/u/Bondo)\
**Post date:** [February 15, 2017, 7:38pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/1 "2017-02-15T19:38:05Z")

</div>

I'm trying to get Logstash to take in an IIS web log and geo locate the IP address using something like GeoLite from MaxMind and then dump it to Elasticsearch. Has anyone done something similar before? I'm not an ELK guru so any help is much appreciated.

I have ELK running, but I can't quite get the syntax in my logstash.conf right in order to get this to work.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 15, 2017, 8:59pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/2 "2017-02-15T20:59:44Z")

</div>

Providing what you do have would be helpful to get started.

---

<div class="post-metadata">

**Author:** ![Bondo](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@Bondo](https://discuss.elastic.co/u/Bondo)\
**Post date:** [February 15, 2017, 10:21pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/3 "2017-02-15T22:21:47Z")

</div>

I'm trying to keep it simple at the moment, I first want to try and just get my IIS wbelog datat into Elastic. Here is my .conf file

```
input {  
  file {
    type => "iis-w3c"
    path => "C:/logs/*.log"
  }

}

filter {  
  ## Ignore the comments that IIS will add to the start of the W3C logs
  #
  if [message] =~ "^#" {
    drop {}
  }

  grok {
    ## Very helpful site for building these statements:
    # http://grokdebug.herokuapp.com/
    #
    # This is configured to parse out every field of IIS's W3C format when
    # every field is included in the logs
    #
    match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:s-sitename} % {WORD:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} % {NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Cookie)} %{NOTSPACE:cs(Referer)} %{NOTSPACE:cs-host} %{NUMBER:sc-status:int} %{NUMBER:sc- substatus:int} %{NUMBER:sc-win32-status:int} %{NUMBER:sc-bytes:int} %{NUMBER:cs-bytes:int} % {NUMBER:time-taken:int}" ,
"message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:s-sitename} %{WORD:cs-method} % {URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} % {IPORHOST:c-ip} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Referer)} %{NUMBER:response:int} % {NUMBER:sc-substatus:int} %{NUMBER:sc-substatus:int} %{NUMBER:time-taken:int}" ,
"message", "%{TIMESTAMP_ISO8601:timestamp} %{WORD:cs-method} %{URIPATH:cs-uri-stem} % {NOTSPACE:cs-post-data} %{NUMBER:s-port} %{IPORHOST:c-ip} HTTP/%{NUMBER:c-http-version} % {NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Cookie)} %{NOTSPACE:cs(Referer)} %{NOTSPACE:cs- host} %{NUMBER:sc-status:int} %{NUMBER:sc-bytes:int} %{NUMBER:cs-bytes:int} %{NUMBER:time- taken:int}"
]
  }

  ## Set the Event Timesteamp from the log
  #
  date {
    match => ["log_timestamp", "YYYY-MM-dd HH:mm:ss"]
      timezone => "Etc/UTC"
  }

  ## If the log record has a value for 'bytesSent', then add a new field
  # to the event that converts it to kilobytes
  #
  if [bytesSent] {
    ruby {
      code => "event['kilobytesSent'] = event['bytesSent'].to_i / 1024.0"
    }
  }

  ## Do the same conversion for the bytes received value
  #
  if [bytesReceived] {
    ruby {
      code => "event['kilobytesReceived'] = event['bytesReceived'].to_i / 1024.0"
    }
  }

  ## Perform some mutations on the records to prep them for Elastic
  #
  mutate {
    ## Convert some fields from strings to integers
    #
    convert => ["bytesSent", "integer"]
    convert => ["bytesReceived", "integer"]
    convert => ["timetaken", "integer"]

    ## Finally remove the original log_timestamp field since the event will
    # have the proper date on it
    #
    remove_field => ["log_timestamp"]
  }

  ## Parse out the user agent
  #
    useragent {
        source=> "useragent"
        prefix=> "browser"
    }

}

## We're only going to output these records to Elasticsearch so configure
# that.
#
output {  
  elasticsearch {
    embedded => false
    host => "localhost"
    port => 9200
    protocol => "http"
    #
    ## Log records into month-based indexes
    #
    index => "%{type}-%{+YYYY.MM}"
  }

  ## stdout included just for testing
  #
  # stdout {codec => rubydebug}
}
```

---

<div class="post-metadata">

**Author:** ![Bondo](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@Bondo](https://discuss.elastic.co/u/Bondo)\
**Post date:** [February 15, 2017, 10:22pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/4 "2017-02-15T22:22:44Z")

</div>

I don't think it likes my grok though as I get this error when I run it.

```
[ERROR][logstash.agent] fetched an invalid config {:config=>"input { \n file {\n type => \"iis-w3c\"\n path => \"C:/logs/*.log\"\n }\n\n}\n\nfilter { \n ## Ignore the comments that IIS will add to the start of the W3C logs\n #\n if [message] =~ \"^#\" {\n drop {}\n }\n\n grok {\n ## Very helpful site for building these statements:\n # http://grokdebug.herokuapp.com/\n #\n # This is configured to parse out every field of IIS's W3C format when\n # every field is included in the logs\n #\n match => [\"message\", \"%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:s-sitename} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Cookie)} %{NOTSPACE:cs(Referer)} % {NOTSPACE:cs-host} %{NUMBER:sc-status:int} %{NUMBER:sc-substatus:int} %{NUMBER:sc-win32-status:int} %{NUMBER:sc-bytes:int} %{NUMBER:cs-bytes:int} %{NUMBER:time-taken:int}\" ,\n\"message\", \"% {TIMESTAMP_ISO8601:timestamp} %{IPORHOST:s-sitename} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} % {NOTSPACE:cs(User-Agent)} %{NOTSPACE:cs(Referer)} %{NUMBER:response:int} %{NUMBER:sc-substatus:int} %{NUMBER:sc-substatus:int} %{NUMBER:time-taken:int}\" ,\n\"message\", \"% {TIMESTAMP_ISO8601:timestamp} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-post-data} %{NUMBER:s-port} %{IPORHOST:c-ip} HTTP/%{NUMBER:c-http-version} %{NOTSPACE:cs(User- Agent)} %{NOTSPACE:cs(Cookie)} %{NOTSPACE:cs(Referer)} %{NOTSPACE:cs-host} %{NUMBER:sc-status:int} %{NUMBER:sc-bytes:int} %{NUMBER:cs-bytes:int} %{NUMBER:time-taken:int}\"\n]\n }\n\n ## Set the Event Timesteamp from the log\n #\n date {\n match => [\"log_timestamp\", \"YYYY-MM-dd HH:mm:ss\"]\n timezone => \"Etc/UTC\"\n }\n\n ## If the log record has a value for 'bytesSent', then add a new field\n # to the event that converts it to kilobytes\n #\n if [bytesSent] {\n ruby {\n code => \"event['kilobytesSent'] = event['bytesSent'].to_i / 1024.0\"\n }\n }\n\n\n ## Do the same conversion for the bytes received value\n #\n if [bytesReceived] {\n ruby {\n code => \"event['kilobytesReceived'] = event['bytesReceived'].to_i / 1024.0\"\n }\n }\n\n ## Perform some mutations on the records to prep them for Elastic\n #\n mutate {\n ## Convert some fields from strings to integers\n #\n convert => [\"bytesSent\", \"integer\"]\n convert => [\"bytesReceived\", \"integer\"]\n convert => [\"timetaken\", \"integer\"]\n\n ## Finally remove the original log_timestamp field since the event will\n # have the proper date on it\n #\n remove_field => [\"log_timestamp\"]\n }\n\n ## Parse out the user agent\n #\n useragent {\n source=> \"useragent\"\n prefix=> \"browser\"\n }\n\n}\n\n## We're only going to output these records to Elasticsearch so configure\n# that.\n#\noutput { \n elasticsearch {\n embedded => false\n host => \"localhost\"\n port => 9200\n protocol => \"http\"\n #\n ## Log records into month-based indexes\n #\n index => \"%{type}-%{+YYYY.MM}\"\n }\n\n ## stdout included just for testing\n #\n # stdout {codec => rubydebug}\n}\n", :reason=>"Something is wrong with your configuration."}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 15, 2017, 10:49pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/5 "2017-02-15T22:49:59Z")

</div>

> [@Bondo](#):
>
> % {WORD:cs-method}

You have a few instances of that you should fix.

Also I cannot see an geoip filter.

---

<div class="post-metadata">

**Author:** ![Bondo](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@Bondo](https://discuss.elastic.co/u/Bondo)\
**Post date:** [February 16, 2017, 1:59am UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/6 "2017-02-16T01:59:40Z")

</div>

N00b here, what do I need to fix with the %{WORD:cs-method} references?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 16, 2017, 2:06am UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/7 "2017-02-16T02:06:54Z")

</div>

The config you posted has that.

---

<div class="post-metadata">

**Author:** ![Bondo](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@Bondo](https://discuss.elastic.co/u/Bondo)\
**Post date:** [February 16, 2017, 2:53am UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/8 "2017-02-16T02:53:01Z")

</div>

I see it in there, not sure what I need to fix about them though.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 16, 2017, 2:55am UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/9 "2017-02-16T02:55:31Z")

</div>

The space between `%` and `{` is invalid.

---

<div class="post-metadata">

**Author:** ![Bondo](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@Bondo](https://discuss.elastic.co/u/Bondo)\
**Post date:** [February 16, 2017, 3:09am UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/10 "2017-02-16T03:09:25Z")

</div>

Must be just the way I entered it into this forum post. My actual conf file doesn't have the space.

---

<div class="post-metadata">

**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\
**Post date:** [February 16, 2017, 4:38am UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/11 "2017-02-16T04:38:31Z")

</div>

This is how I process IIS logs [http://www.sysad.me/2016/07/elastic-stack-process-iis-logs.html](http://www.sysad.me/2016/07/elastic-stack-process-iis-logs.html). BTW, why do you need to do this kilobytes converting?

> [@Bondo](#):
>
> ## If the log record has a value for 'bytesSent', then add a new field
> 
> # to the event that converts it to kilobytes
> 
> # 
> 
> if [bytesSent] {  
> ruby {  
> code =\> "event['kilobytesSent'] = event['bytesSent'].to\_i / 1024.0"  
> }  
> }
> 
> ## Do the same conversion for the bytes received value
> 
> # 
> 
> if [bytesReceived] {  
> ruby {  
> code =\> "event['kilobytesReceived'] = event['bytesReceived'].to\_i / 1024.0"  
> }  
> }

If you want LS to process IIS Logs, Why don't you use CSV filter plugin?

---

<div class="post-metadata">

**Author:** ![Bondo](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@Bondo](https://discuss.elastic.co/u/Bondo)\
**Post date:** [February 16, 2017, 6:41pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/12 "2017-02-16T18:41:20Z")

</div>

I've modified my conf now a little and am able to get IIS logs into Elastic. I'm not sure what to do next with getting the geolocation based on IP though. From what I understand, Logstash has the MaxMind geo database built into it already? Does anyone have an example how to get this working?

```
input {
 file {
   	#type => "iis"
    path => "C:/logs/*.log"
    start_position => "beginning" 
  }
}

filter {

  #ignore log comments
  if [message] =~ "^#" {
    drop {}
  }

  grok {
    # check that fields match your IIS log settings
    match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:s-sitename} %{IPORHOST:s-ip} %{WORD:cs-method} %{URIPATH:cs-uri-stem} %{NOTSPACE:cs-uri-query} %{NUMBER:s-port} %{NOTSPACE:cs-username} %{IPORHOST:c-ip} %{WORD:cs-version}"]
  }

  #Set the Event Timesteamp from the log
    date {
    match => ["log_timestamp", "YYYY-MM-dd HH:mm:ss"]
    timezone => "Etc/UTC"
  }	

  mutate {
	remove_field => ["log_timestamp"]
  }
}

# See documentation for different protocols:
# http://logstash.net/docs/1.4.2/outputs/elasticsearch
output {
  # stdout { codec => rubydebug }
    elasticsearch { hosts => ["localhost:9200"] }
}
```

---

<div class="post-metadata">

**Author:** ![anhlqn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anhlqn/32/5454_2.png) [@anhlqn](https://discuss.elastic.co/u/anhlqn)\
**Post date:** [February 16, 2017, 7:11pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/13 "2017-02-16T19:11:46Z")

</div>

> [@anhlqn](#):
>
> This is how I process IIS logs [http://www.sysad.me/2016/07/elastic-stack-process-iis-logs.html](http://www.sysad.me/2016/07/elastic-stack-process-iis-logs.html)

It's in there 😃

---

<div class="post-metadata">

**Author:** ![Bondo](https://avatars.discourse-cdn.com/v4/letter/b/ad7895/32.png) [@Bondo](https://discuss.elastic.co/u/Bondo)\
**Post date:** [February 17, 2017, 6:47pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/14 "2017-02-17T18:47:16Z")

</div>

Thanks anhlqn, I'll have to check that out. I was able to get this to work now successfully.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 17, 2017, 6:47pm UTC](https://discuss.elastic.co/t/logstash-geolocate-ip-address-into-elasticsearch/75246/15 "2017-03-17T18:47:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.