# Logstash Setup with GeoIP

**URL:** <https://discuss.elastic.co/t/logstash-setup-with-geoip/195231>\
**Category:** Logstash\
**Created:** [August 14, 2019, 3:34pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231 "2019-08-14T15:34:34Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [August 14, 2019, 3:34pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/1 "2019-08-14T15:34:34Z")

</div>

I am attempting to setup geoip on Logstash 7.3.0. I have seen an old post on how to set it up, but I want to make sure I'm doing it right. In the logstash config file I add the below code. Is this correct?

```
filter {
geoip {
source => "source.ip"
target => "src_geoip"
}
geoip {
source => "destination.ip"
target => "dst_geoip"
}
}

```

Thanks in advance

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [August 16, 2019, 1:23am UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/2 "2019-08-16T01:23:01Z")

</div>

Any advice would be appreciated.

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [August 21, 2019, 7:50pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/3 "2019-08-21T19:50:49Z")

</div>

Let me ask this a different way. I have private IPs I want to enrich with data, but not sure to achieve this. Any guidance would be grateful.

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [September 9, 2019, 3:27pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/4 "2019-09-09T15:27:04Z")

</div>

Can someone give me advice to get started?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 9, 2019, 3:40pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/5 "2019-09-09T15:40:50Z")

</div>

> [@Marcell0e](#):
>
> source =\> "source.ip"

If your field name contains a period then this is correct. If the source object contains a field called ip then you should use "[source][ip]".

To have the fields [src\_geoip][location] be a geo\_point you will need an index template that tells elasticsearch that.

If by "private" IPs you mean addresses in the blocks reserved by RFC 1918 then you will have to build your own database in which to look them up.

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [September 9, 2019, 3:48pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/6 "2019-09-09T15:48:59Z")

</div>

@Badger thank you for your response. I have a couple of followup questions.

> To have the fields [src\_geoip][location] be a geo\_point you will need an index template that tells elasticsearch that.

I thought I read that the default index did the geo\_point?

Can I set up something like below in my Logstash config file in the filter section?

```
	if [source.ip] =~ /^10\.3\./ {
   mutate { replace => { "[geoip][timezone]" => "Eastern/New York" } }
   mutate { replace => { "[geoip][country_name]" => "Office" } }
   mutate { replace => { "[geoip][country_code2]" => "O" } }
   mutate { replace => { "[geoip][country_code3]" => "O" } }
   mutate { remove_field => ["[geoip][location]" ] }
   mutate { add_field => { "[geoip][location]" => "-106.158" } }
   mutate { add_field => { "[geoip][location]" => "109.768" } }
   mutate { convert => ["[geoip][location]", "float" ] }
   mutate { replace => ["[geoip][latitude]", 109.768 ] }
   mutate { convert => ["[geoip][latitude]", "float" ] }
   mutate { replace => ["[geoip][longitude]", -106.158 ] }
   mutate { convert => ["[geoip][longitude]", "float" ] }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 9, 2019, 4:02pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/7 "2019-09-09T16:02:51Z")

</div>

The [default](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/77b48f70e244f69711b732e95d554e837a1c604a/lib/logstash/outputs/elasticsearch/elasticsearch-template-es5x.json#L34) template for an elasticsearch output defines the location field of an object called geoip to be a geo\_point. That matches the default target for the geoip filter. If you use a different target then you need a different template.

You could build a geoip object like that, yes.

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [September 9, 2019, 4:07pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/8 "2019-09-09T16:07:32Z")

</div>

When i try to implement that into the Logstash config file I get a Signt error and the main pipleline stops. I can't seem to find how to do this on 7.3.0. What am i doing wrong?

Thank you for your help.

---

<div class="post-metadata">

**Author:** ![lcer00](https://avatars.discourse-cdn.com/v4/letter/l/d2c977/32.png) [@lcer00](https://discuss.elastic.co/u/lcer00)\
**Post date:** [September 9, 2019, 4:12pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/9 "2019-09-09T16:12:28Z")

</div>

> [@Badger](#):
>
> object called geoip

Hallo,

if you are using filebeat you should look at [Geo Fields | Elastic Common Schema (ECS) Reference [8.11] | Elastic](https://www.elastic.co/guide/en/ecs/current/ecs-geo.html)

the new filebeat index template (dont know how long, for me 7.3.0) does not use geoip.location but geo.location as its geo\_point field.

so you may use something like  
target =\> "[destination][geo]"

and the field will fit the template.

lcer

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [September 9, 2019, 4:19pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/10 "2019-09-09T16:19:52Z")

</div>

So, my Logstash config filter section should look like this then?

```
filter {
   geoip {
       source=>"source.ip"
       target=>"[destination][geo]"
            }
  	if [source.ip] =~ /^10\.3\./ {
   mutate { replace => { "[geoip][timezone]" => "Eastern/New York" } }
   mutate { replace => { "[geoip][country_name]" => "Office" } }
   mutate { replace => { "[geoip][country_code2]" => "O" } }
   mutate { replace => { "[geoip][country_code3]" => "O" } }
   mutate { remove_field => ["[geoip][location]" ] }
   mutate { add_field => { "[geoip][location]" => "-106.158" } }
   mutate { add_field => { "[geoip][location]" => "109.768" } }
   mutate { convert => ["[geoip][location]", "float" ] }
   mutate { replace => ["[geoip][latitude]", 109.768 ] }
   mutate { convert => ["[geoip][latitude]", "float" ] }
   mutate { replace => ["[geoip][longitude]", -106.158 ] }
   mutate { convert => ["[geoip][longitude]", "float" ] }
   }
}
```

---

<div class="post-metadata">

**Author:** ![lcer00](https://avatars.discourse-cdn.com/v4/letter/l/d2c977/32.png) [@lcer00](https://discuss.elastic.co/u/lcer00)\
**Post date:** [September 9, 2019, 5:18pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/11 "2019-09-09T17:18:43Z")

</div>

Hallo,

1. you should use [geo] instead of [geoip]

2. if source.ip is a nested field you should address is as [source][ip]

3. you should change your if structure. If the geoip filter tries to resolve a local address, it fails. In this case none of the [geo/geoip][...] fields will exists

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [September 9, 2019, 6:01pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/12 "2019-09-09T18:01:30Z")

</div>

Thank you for all your help. Does this look right? I'm new to this and learning as I go. I figure a lot out by trial and error (inference on error). Another dumb question. How do I know if it is nested? I'm using packetbeat for the IP and one of the categories is source.ip.

```
if [source][ip] =~ /^10.3./ {
   mutate { replace => { "[geo][timezone]" => "Eastern/New York" } }
   mutate { replace => { "[geo][country_name]" => "Office" } }
   mutate { replace => { "[geo][country_code2]" => "O" } }
   mutate { replace => { "[geo][country_code3]" => "O" } }
   mutate { remove_field => ["[geo][location]" ] }
   mutate { add_field => { "[geo][location]" => "-106.158" } }
   mutate { add_field => { "[geo][location]" => "109.768" } }
   mutate { convert => ["[geo][location]", "float" ] }
   mutate { replace => ["[geo][latitude]", 109.768 ] }
   mutate { convert => ["[geo][latitude]", "float" ] }
   mutate { replace => ["[geo][longitude]", -106.158 ] }
   mutate { convert => ["[geo][longitude]", "float" ] }
}
} else {
geoip {
source => "[source][ip]"
target=>"[destination][geo]"
}
}
```

---

<div class="post-metadata">

**Author:** ![lcer00](https://avatars.discourse-cdn.com/v4/letter/l/d2c977/32.png) [@lcer00](https://discuss.elastic.co/u/lcer00)\
**Post date:** [September 9, 2019, 6:33pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/13 "2019-09-09T18:33:20Z")

</div>

Looks Good but:

Replace does not create fields. But the fields do not exists until created. The geoip Filter creates this fields if successful. And you can use mutate add\_field to create them.

lcer

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [September 9, 2019, 6:56pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/14 "2019-09-09T18:56:01Z")

</div>

How would I go about creating them first time around? Would I use create instead of replace? Once they are created I can change the create to replace? Not sure how to address that issue.

---

<div class="post-metadata">

**Author:** ![lcer00](https://avatars.discourse-cdn.com/v4/letter/l/d2c977/32.png) [@lcer00](https://discuss.elastic.co/u/lcer00)\
**Post date:** [September 9, 2019, 8:27pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/15 "2019-09-09T20:27:16Z")

</div>

Well, You misunderstand elasticsearch.  
You do not create fields for the elasticsearch „table“. You have to create fields in each document. You can store very small documents containing only a few fields in the same elasticsearch index together with documents containing hundreds of fields. But if a field is not inside the document you have to create it ( e.g. Mutate add\_field ) before you can use it.

lcer

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [September 10, 2019, 2:01pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/16 "2019-09-10T14:01:41Z")

</div>

I'm trying to understand elasticsearch a little bit every day. Is there a way to see which fields are already created? Is there a better way to create the fields before I implement the Logstash filter?

Thank you for helping me understand.

---

<div class="post-metadata">

**Author:** ![lcer00](https://avatars.discourse-cdn.com/v4/letter/l/d2c977/32.png) [@lcer00](https://discuss.elastic.co/u/lcer00)\
**Post date:** [September 10, 2019, 7:50pm UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/17 "2019-09-10T19:50:57Z")

</div>

Hallo

elasticsearch stores documents.  
Every event in Logstash is one document.  
A document may have several fields - elasticsearch does not need to know which fields. You can create any field inside a logstash event - elasticsearch will store it. If every event should have a field called „test\_data“ you have to create it in every event seperately. You cannot create it „one time for all documents“ - not in logstash - not in elasticsearch.

lcer

---

<div class="post-metadata">

**Author:** ![Marcell0e](https://avatars.discourse-cdn.com/v4/letter/m/ebca7d/32.png) [@Marcell0e](https://discuss.elastic.co/u/Marcell0e)\
**Post date:** [September 11, 2019, 2:58am UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/18 "2019-09-11T02:58:02Z")

</div>

If I wanted to add another different office would I add another if statement? Something like below.

```
if [source][ip] =~ /^10.3./ {
...
}
} else {
if [source][ip] =~ /^10.4./ {
...
}
} else {
geoip {
source => "[source][ip]"
target=>"[destination][geo]"
}
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2019, 3:13am UTC](https://discuss.elastic.co/t/logstash-setup-with-geoip/195231/19 "2019-10-09T03:13:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.