Private ip geoip JSON object / structure and some questions

I've seen a bunch of topics on how to map your private ip address to a geoip json object but I have a question on the fields and the structure. I'm following this thread Creating geoip data for internal networks but might switch over to this Create Custom geoip database for Logstash 5.2 as I have many private ip's I need to map

EDIT: link to second option: Private ip geoip from dictionary

Question 1 is about the location, do I need to have the nested [lat] and [lon] as in example #2 or is example #1 correct?

Question 2 is, do I need the [geo][ip] line? I have this already in [apache][access][remote_ip] and I wanted to know if this is required for the map in the dashboard

Question 3, I see the geo object in kibana but when I use the apache dashboard it does not plot the map. is there something else I need to do?

Question 4, where do I find the value for region_code?

Example: #1

{
	"geoip": {
		"timezone": "America/Detroit",
		"continent_code": "US",
		"country_name": "United States",
		"region_code": "1111111111111",
		"country_code2": "US",
		"country_code3": "US",
		"region_name": "Michigan",
		"ip": "220.181.108.103",
		"city_name": "Detroit",
		"latitude": 42.5597,
		"longitude": -83.1138,
		"location": [42.5597, -83.1138]
	}
}

Example #2

{
	"geoip": {
		"timezone": "America/Detroit",
		"continent_code": "US",
		"country_name": "United States",
		"region_code": "1111111111111",
		"country_code2": "US",
		"country_code3": "US",
		"region_name": "Michigan",
		"ip": "220.181.108.103",
		"city_name": "Detroit",
		"latitude": 42.5597,
		"longitude": -83.1138,
		"location": {
			"lon": -83.1138,
			"lat": 42.5597
		}
	}
}

partial logstash config

if [apache2][access][remote_ip] =~ /^220.181.108.*/ {
    mutate { 
      replace => { 
        "[geoip][timezone]" => "America/Detroit" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][continent_code]" => "US" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][country_name]" => "United States" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][region_code]" => "MI" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][country_code2]" => "US" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][country_code3]" => "US" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][region_name]" => "Michigan" 
      } 
    } 
    mutate { 
      remove_field => [ "[geoip][location]" ] 
    } 
    mutate { 
      add_field => { 
        "[geoip][location]" => "-83.1138" 
      } 
    } 
    mutate { 
      add_field => { 
        "[geoip][location]" => "42.5597" 
      } 
    } 
    mutate { 
      convert => [ "[geoip][location]","float" ] 
    } 
    mutate { 
      replace => [ "[geoip][latitude]","42.5597" ] 
    } 
    mutate { 
      convert => [ "[geoip][latitude]","float" ] 
    } 
    mutate { 
      replace => [ "[geoip][longitude]","-83.1138" ] 
    } 
    mutate { 
      convert => [ "[geoip][longitude]","float" ] 
    } 
  }
  1. Either should work, as long as it's mapped correctly.
  2. Nope, just the merged lat+lon field
  3. What is the _mapping for the index?
  4. You need to add it

Side Note: Im using Fliebeat Apache Module and updating the index from filebeat-* to logstash-* (This is working) just wanted to add this info just in case this changes how to proceed

from https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html

GET /_all/_mapping

output

"geoip": {
"properties": {
	"city_name": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"continent_name": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"country_iso_code": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"location": {
		"type": "geo_point"
	},
	"region_iso_code": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"region_name": {
		"type": "keyword",
		"ignore_above": 1024
	}
}

also follow up on the location, does lat or lon need to go first? I saw this post Creating geoip data for internal networks but wanted to confirm

Im not sure what else Im missing to see them be in the map view

Hmm I think I see the issue but not clear on how to fix it

I see the geoip object but not in the correct spot?

 "geoip": {
 	"region_code": "AK",
 	"country_name": "United States",
 	"country_code2": "US",
 	"location": [
 		61.19,
 		-149.8938
 	],
 	"longitude": -149.8938,
 	"country_code3": "US",
 	"timezone": "America/Anchorage",
 	"ip": "220.181.108.103",
 	"continent_code": "US",
 	"city_name": "Anchorage",
 	"region_name": "Alaska",
 	"latitude": 61.19
 },

and I also see the geoip in the apache object but with no data

"apache2": {
	"access": {
		"referrer": "-",
		"geoip": {},
		"http_version": "1.1",
		"body_sent": {
			"bytes": "443"
		},
		"user_agent": {
			"name": "Other",
			"build": "",
			"os": "Other",
			"os_name": "Other",
			"device": "Other"
		},
		"remote_ip": "220.181.108.103",
		"response_code": "200",
		"url": "/version/",
		"method": "GET",
		"user_name": "-"
	}
}

how do I move the whole object to the correct location?

The mappings needs to be like this - https://www.elastic.co/guide/en/elasticsearch/reference/6.4/geo-point.html. So as long as it is in one of those formats you are ok.

Can you show the entire mapping for the index, as well as a sample doc? Feel free to use gist/pastebin/etc.

here is the gist https://gist.github.com/phillpafford/15caf428f8a3dc37e37d3dc1f2a469ef it's rather large

ok so using this command ( GET myindex/_mapping )

GET logstash-2018.11/_mapping

I see the location is float

"geoip": {
  "properties": {
    "city_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "continent_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "country_code2": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "country_code3": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "country_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "ip": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "latitude": {
      "type": "float"
    },
    "location": {
      "type": "float"
    },
    "longitude": {
      "type": "float"
    },
    "region_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "region_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "timezone": {
      "type": "keyword",
      "ignore_above": 1024
    }
  }
}

when I try

PUT logstash-2018.11
{
  "mappings": {
    "doc": {
      "properties": {
        "geoip.location": {
          "type": "geo_point"
        }
      }
    }
  }
}

I get this exception:

resource_already_exists_exception

Yep, you need to apply the mapping via a template, or to an empty index. You cannot apply it retroactively.

thanks but Im at a loss. can I not update the existing index/template? if not, how do I go about this in the existing index or can this not be done? if I add a new index, does the data need to be in this index?

You can apply it to the existing template. Just make sure you update it with the new mapping and replace the entire thing, not just that section.

Then you may just want to wait till a new index is created (ie tomorrow).

ok still stumbling around, can you point me to the docs? maybe a working example as well?

The idea is to replace the entire existing template;

  1. GET _template/template_name
  2. Update the template and add your section
  3. POST _template/template_name

ok I ran

GET _template/logstash

and then with the JSON and added geoip section

POST _template/logstash
{
    ... Same JSON from get but with added section for GEOIP at the same level as the apache section ...
}

example

"nginx": {
},
"geoip": {
  "properties": {
    "country_iso_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "location": {
      "type": "geo_point"
    },
    "region_iso_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "continent_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "city_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "region_name": {
      "type": "keyword",
      "ignore_above": 1024
    }
  }
},
"apache2": {
}

but I get this error

{
  "error": {
    "root_cause": [
      {
        "type": "action_request_validation_exception",
        "reason": "Validation Failed: 1: index patterns are missing;"
      }
    ],
    "type": "action_request_validation_exception",
    "reason": "Validation Failed: 1: index patterns are missing;"
  },
  "status": 400
}

I'd have to see the entire thing you are posting sorry.

solved this here Private ip geoip from dictionary

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.