# Private ip geoip JSON object / structure and some questions

**URL:** <https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555>\
**Category:** Logstash\
**Created:** [November 13, 2018, 10:21pm UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555 "2018-11-13T22:21:21Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![ppafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppafford/32/36533_2.png) [@ppafford](https://discuss.elastic.co/u/ppafford)\
**Post date:** [November 13, 2018, 10:21pm UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/1 "2018-11-13T22:21:21Z")

</div>

I've seen a bunch of topics on how to map your private ip address to a geoip json object but I have a question on the fields and the structure. I'm following this thread [Creating geoip data for internal networks](https://discuss.elastic.co/t/creating-geoip-data-for-internal-networks/729/10) but might switch over to this [Create Custom geoip database for Logstash 5.2](https://discuss.elastic.co/t/create-custom-geoip-database-for-logstash-5-2/79473/17) as I have many private ip's I need to map

EDIT: link to second option: [Private ip geoip from dictionary](https://discuss.elastic.co/t/private-ip-geoip-from-dictionary/156763)

Question 1 is about the location, do I need to have the nested [lat] and [lon] as in example #2 or is example #1 correct?

Question 2 is, do I need the [geo][ip] line? I have this already in [apache][access][remote\_ip] and I wanted to know if this is required for the map in the dashboard

Question 3, I see the geo object in kibana but when I use the apache dashboard it does not plot the map. is there something else I need to do?

Question 4, where do I find the value for region\_code?

Example: #1

```
{
	"geoip": {
		"timezone": "America/Detroit",
		"continent_code": "US",
		"country_name": "United States",
		"region_code": "1111111111111",
		"country_code2": "US",
		"country_code3": "US",
		"region_name": "Michigan",
		"ip": "220.181.108.103",
		"city_name": "Detroit",
		"latitude": 42.5597,
		"longitude": -83.1138,
		"location": [42.5597, -83.1138]
	}
}

```

Example #2

```
{
	"geoip": {
		"timezone": "America/Detroit",
		"continent_code": "US",
		"country_name": "United States",
		"region_code": "1111111111111",
		"country_code2": "US",
		"country_code3": "US",
		"region_name": "Michigan",
		"ip": "220.181.108.103",
		"city_name": "Detroit",
		"latitude": 42.5597,
		"longitude": -83.1138,
		"location": {
			"lon": -83.1138,
			"lat": 42.5597
		}
	}
}

```

partial logstash config

```
if [apache2][access][remote_ip] =~ /^220.181.108.*/ {
    mutate { 
      replace => { 
        "[geoip][timezone]" => "America/Detroit" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][continent_code]" => "US" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][country_name]" => "United States" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][region_code]" => "MI" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][country_code2]" => "US" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][country_code3]" => "US" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][region_name]" => "Michigan" 
      } 
    } 
    mutate { 
      remove_field => ["[geoip][location]" ] 
    } 
    mutate { 
      add_field => { 
        "[geoip][location]" => "-83.1138" 
      } 
    } 
    mutate { 
      add_field => { 
        "[geoip][location]" => "42.5597" 
      } 
    } 
    mutate { 
      convert => ["[geoip][location]","float" ] 
    } 
    mutate { 
      replace => ["[geoip][latitude]","42.5597" ] 
    } 
    mutate { 
      convert => ["[geoip][latitude]","float" ] 
    } 
    mutate { 
      replace => ["[geoip][longitude]","-83.1138" ] 
    } 
    mutate { 
      convert => ["[geoip][longitude]","float" ] 
    } 
  }
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 14, 2018, 1:46am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/2 "2018-11-14T01:46:27Z")

</div>

1. Either should work, as long as it's mapped correctly.
2. Nope, just the merged lat+lon field
3. What is the `_mapping` for the index?
4. You need to add it

---

<div class="post-metadata">

**Author:** ![ppafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppafford/32/36533_2.png) [@ppafford](https://discuss.elastic.co/u/ppafford)\
**Post date:** [November 14, 2018, 2:47am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/3 "2018-11-14T02:47:01Z")

</div>

Side Note: Im using Fliebeat Apache Module and updating the index from filebeat-\* to logstash-\* (This is working) just wanted to add this info just in case this changes how to proceed

from [https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html)

```
GET /_all/_mapping

```

output

```
"geoip": {
"properties": {
	"city_name": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"continent_name": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"country_iso_code": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"location": {
		"type": "geo_point"
	},
	"region_iso_code": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"region_name": {
		"type": "keyword",
		"ignore_above": 1024
	}
}

```

also follow up on the location, does lat or lon need to go first? I saw this post [Creating geoip data for internal networks](https://discuss.elastic.co/t/creating-geoip-data-for-internal-networks/729/2?u=ppafford) but wanted to confirm

Im not sure what else Im missing to see them be in the map view

---

<div class="post-metadata">

**Author:** ![ppafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppafford/32/36533_2.png) [@ppafford](https://discuss.elastic.co/u/ppafford)\
**Post date:** [November 14, 2018, 3:40am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/4 "2018-11-14T03:40:02Z")

</div>

Hmm I think I see the issue but not clear on how to fix it

I see the geoip object but not in the correct spot?

```
 "geoip": {
 	"region_code": "AK",
 	"country_name": "United States",
 	"country_code2": "US",
 	"location": [
 		61.19,
 		-149.8938
 	],
 	"longitude": -149.8938,
 	"country_code3": "US",
 	"timezone": "America/Anchorage",
 	"ip": "220.181.108.103",
 	"continent_code": "US",
 	"city_name": "Anchorage",
 	"region_name": "Alaska",
 	"latitude": 61.19
 },

```

and I also see the geoip in the apache object but with no data

```
"apache2": {
	"access": {
		"referrer": "-",
		"geoip": {},
		"http_version": "1.1",
		"body_sent": {
			"bytes": "443"
		},
		"user_agent": {
			"name": "Other",
			"build": "",
			"os": "Other",
			"os_name": "Other",
			"device": "Other"
		},
		"remote_ip": "220.181.108.103",
		"response_code": "200",
		"url": "/version/",
		"method": "GET",
		"user_name": "-"
	}
}

```

how do I move the whole object to the correct location?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 14, 2018, 4:26am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/5 "2018-11-14T04:26:59Z")

</div>

The mappings needs to be like this - [https://www.elastic.co/guide/en/elasticsearch/reference/6.4/geo-point.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/geo-point.html). So as long as it is in one of those formats you are ok.

Can you show the entire mapping for the index, as well as a sample doc? Feel free to use gist/pastebin/etc.

---

<div class="post-metadata">

**Author:** ![ppafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppafford/32/36533_2.png) [@ppafford](https://discuss.elastic.co/u/ppafford)\
**Post date:** [November 14, 2018, 3:34pm UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/6 "2018-11-14T15:34:50Z")

</div>

here is the gist [https://gist.github.com/phillpafford/15caf428f8a3dc37e37d3dc1f2a469ef](https://gist.github.com/phillpafford/15caf428f8a3dc37e37d3dc1f2a469ef) it's rather large

---

<div class="post-metadata">

**Author:** ![ppafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppafford/32/36533_2.png) [@ppafford](https://discuss.elastic.co/u/ppafford)\
**Post date:** [November 14, 2018, 5:10pm UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/7 "2018-11-14T17:10:33Z")

</div>

ok so using this command ( GET myindex/\_mapping )

```
GET logstash-2018.11/_mapping

```

I see the location is float

```
"geoip": {
  "properties": {
    "city_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "continent_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "country_code2": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "country_code3": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "country_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "ip": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "latitude": {
      "type": "float"
    },
    "location": {
      "type": "float"
    },
    "longitude": {
      "type": "float"
    },
    "region_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "region_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "timezone": {
      "type": "keyword",
      "ignore_above": 1024
    }
  }
}

```

when I try

```
PUT logstash-2018.11
{
  "mappings": {
    "doc": {
      "properties": {
        "geoip.location": {
          "type": "geo_point"
        }
      }
    }
  }
}

```

I get this exception:

```
resource_already_exists_exception
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 14, 2018, 11:05pm UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/8 "2018-11-14T23:05:28Z")

</div>

Yep, you need to apply the mapping via a template, or to an empty index. You cannot apply it retroactively.

---

<div class="post-metadata">

**Author:** ![ppafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppafford/32/36533_2.png) [@ppafford](https://discuss.elastic.co/u/ppafford)\
**Post date:** [November 14, 2018, 11:12pm UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/9 "2018-11-14T23:12:39Z")

</div>

thanks but Im at a loss. can I not update the existing index/template? if not, how do I go about this in the existing index or can this not be done? if I add a new index, does the data need to be in this index?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 14, 2018, 11:16pm UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/10 "2018-11-14T23:16:02Z")

</div>

You can apply it to the existing template. Just make sure you update it with the new mapping and replace the entire thing, not just that section.

Then you may just want to wait till a new index is created (ie tomorrow).

---

<div class="post-metadata">

**Author:** ![ppafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppafford/32/36533_2.png) [@ppafford](https://discuss.elastic.co/u/ppafford)\
**Post date:** [November 14, 2018, 11:48pm UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/11 "2018-11-14T23:48:04Z")

</div>

ok still stumbling around, can you point me to the docs? maybe a working example as well?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 15, 2018, 1:28am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/12 "2018-11-15T01:28:20Z")

</div>

The idea is to replace the entire existing template;

1. `GET _template/template_name`
2. Update the template and add your section
3. `POST _template/template_name`

---

<div class="post-metadata">

**Author:** ![ppafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppafford/32/36533_2.png) [@ppafford](https://discuss.elastic.co/u/ppafford)\
**Post date:** [November 15, 2018, 2:54am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/13 "2018-11-15T02:54:52Z")

</div>

ok I ran

```
GET _template/logstash

```

and then with the JSON and added geoip section

```
POST _template/logstash
{
    ... Same JSON from get but with added section for GEOIP at the same level as the apache section ...
}

```

example

```
"nginx": {
},
"geoip": {
  "properties": {
    "country_iso_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "location": {
      "type": "geo_point"
    },
    "region_iso_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "continent_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "city_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "region_name": {
      "type": "keyword",
      "ignore_above": 1024
    }
  }
},
"apache2": {
}

```

but I get this error

```
{
  "error": {
    "root_cause": [
      {
        "type": "action_request_validation_exception",
        "reason": "Validation Failed: 1: index patterns are missing;"
      }
    ],
    "type": "action_request_validation_exception",
    "reason": "Validation Failed: 1: index patterns are missing;"
  },
  "status": 400
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 15, 2018, 3:53am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/14 "2018-11-15T03:53:41Z")

</div>

I'd have to see the entire thing you are posting sorry.

---

<div class="post-metadata">

**Author:** ![ppafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppafford/32/36533_2.png) [@ppafford](https://discuss.elastic.co/u/ppafford)\
**Post date:** [December 4, 2018, 3:32am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/15 "2018-12-04T03:32:30Z")

</div>

solved this here [Private ip geoip from dictionary](https://discuss.elastic.co/t/private-ip-geoip-from-dictionary/156763/7?u=ppafford)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2019, 3:32am UTC](https://discuss.elastic.co/t/private-ip-geoip-json-object-structure-and-some-questions/156555/16 "2019-01-01T03:32:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.