Private ip geoip JSON object / structure and some questions


(Phill Pafford) #1

I've seen a bunch of topics on how to map your private ip address to a geoip json object but I have a question on the fields and the structure. I'm following this thread Creating geoip data for internal networks but might switch over to this Create Custom geoip database for Logstash 5.2 as I have many private ip's I need to map

EDIT: link to second option: Private ip geoip from dictionary

Question 1 is about the location, do I need to have the nested [lat] and [lon] as in example #2 or is example #1 correct?

Question 2 is, do I need the [geo][ip] line? I have this already in [apache][access][remote_ip] and I wanted to know if this is required for the map in the dashboard

Question 3, I see the geo object in kibana but when I use the apache dashboard it does not plot the map. is there something else I need to do?

Question 4, where do I find the value for region_code?

Example: #1

{
	"geoip": {
		"timezone": "America/Detroit",
		"continent_code": "US",
		"country_name": "United States",
		"region_code": "1111111111111",
		"country_code2": "US",
		"country_code3": "US",
		"region_name": "Michigan",
		"ip": "220.181.108.103",
		"city_name": "Detroit",
		"latitude": 42.5597,
		"longitude": -83.1138,
		"location": [42.5597, -83.1138]
	}
}

Example #2

{
	"geoip": {
		"timezone": "America/Detroit",
		"continent_code": "US",
		"country_name": "United States",
		"region_code": "1111111111111",
		"country_code2": "US",
		"country_code3": "US",
		"region_name": "Michigan",
		"ip": "220.181.108.103",
		"city_name": "Detroit",
		"latitude": 42.5597,
		"longitude": -83.1138,
		"location": {
			"lon": -83.1138,
			"lat": 42.5597
		}
	}
}

partial logstash config

if [apache2][access][remote_ip] =~ /^220.181.108.*/ {
    mutate { 
      replace => { 
        "[geoip][timezone]" => "America/Detroit" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][continent_code]" => "US" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][country_name]" => "United States" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][region_code]" => "MI" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][country_code2]" => "US" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][country_code3]" => "US" 
      } 
    } 
    mutate { 
      replace => { 
        "[geoip][region_name]" => "Michigan" 
      } 
    } 
    mutate { 
      remove_field => [ "[geoip][location]" ] 
    } 
    mutate { 
      add_field => { 
        "[geoip][location]" => "-83.1138" 
      } 
    } 
    mutate { 
      add_field => { 
        "[geoip][location]" => "42.5597" 
      } 
    } 
    mutate { 
      convert => [ "[geoip][location]","float" ] 
    } 
    mutate { 
      replace => [ "[geoip][latitude]","42.5597" ] 
    } 
    mutate { 
      convert => [ "[geoip][latitude]","float" ] 
    } 
    mutate { 
      replace => [ "[geoip][longitude]","-83.1138" ] 
    } 
    mutate { 
      convert => [ "[geoip][longitude]","float" ] 
    } 
  }

Private ip geoip from dictionary
(Mark Walkom) #2
  1. Either should work, as long as it's mapped correctly.
  2. Nope, just the merged lat+lon field
  3. What is the _mapping for the index?
  4. You need to add it

(Phill Pafford) #3

Side Note: Im using Fliebeat Apache Module and updating the index from filebeat-* to logstash-* (This is working) just wanted to add this info just in case this changes how to proceed

from https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html

GET /_all/_mapping

output

"geoip": {
"properties": {
	"city_name": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"continent_name": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"country_iso_code": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"location": {
		"type": "geo_point"
	},
	"region_iso_code": {
		"type": "keyword",
		"ignore_above": 1024
	},
	"region_name": {
		"type": "keyword",
		"ignore_above": 1024
	}
}

also follow up on the location, does lat or lon need to go first? I saw this post Creating geoip data for internal networks but wanted to confirm

Im not sure what else Im missing to see them be in the map view


(Phill Pafford) #4

Hmm I think I see the issue but not clear on how to fix it

I see the geoip object but not in the correct spot?

 "geoip": {
 	"region_code": "AK",
 	"country_name": "United States",
 	"country_code2": "US",
 	"location": [
 		61.19,
 		-149.8938
 	],
 	"longitude": -149.8938,
 	"country_code3": "US",
 	"timezone": "America/Anchorage",
 	"ip": "220.181.108.103",
 	"continent_code": "US",
 	"city_name": "Anchorage",
 	"region_name": "Alaska",
 	"latitude": 61.19
 },

and I also see the geoip in the apache object but with no data

"apache2": {
	"access": {
		"referrer": "-",
		"geoip": {},
		"http_version": "1.1",
		"body_sent": {
			"bytes": "443"
		},
		"user_agent": {
			"name": "Other",
			"build": "",
			"os": "Other",
			"os_name": "Other",
			"device": "Other"
		},
		"remote_ip": "220.181.108.103",
		"response_code": "200",
		"url": "/version/",
		"method": "GET",
		"user_name": "-"
	}
}

how do I move the whole object to the correct location?


(Mark Walkom) #5

The mappings needs to be like this - https://www.elastic.co/guide/en/elasticsearch/reference/6.4/geo-point.html. So as long as it is in one of those formats you are ok.

Can you show the entire mapping for the index, as well as a sample doc? Feel free to use gist/pastebin/etc.


(Phill Pafford) #6

here is the gist https://gist.github.com/phillpafford/15caf428f8a3dc37e37d3dc1f2a469ef it's rather large


(Phill Pafford) #7

ok so using this command ( GET myindex/_mapping )

GET logstash-2018.11/_mapping

I see the location is float

"geoip": {
  "properties": {
    "city_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "continent_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "country_code2": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "country_code3": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "country_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "ip": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "latitude": {
      "type": "float"
    },
    "location": {
      "type": "float"
    },
    "longitude": {
      "type": "float"
    },
    "region_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "region_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "timezone": {
      "type": "keyword",
      "ignore_above": 1024
    }
  }
}

when I try

PUT logstash-2018.11
{
  "mappings": {
    "doc": {
      "properties": {
        "geoip.location": {
          "type": "geo_point"
        }
      }
    }
  }
}

I get this exception:

resource_already_exists_exception

(Mark Walkom) #8

Yep, you need to apply the mapping via a template, or to an empty index. You cannot apply it retroactively.


(Phill Pafford) #9

thanks but Im at a loss. can I not update the existing index/template? if not, how do I go about this in the existing index or can this not be done? if I add a new index, does the data need to be in this index?


(Mark Walkom) #10

You can apply it to the existing template. Just make sure you update it with the new mapping and replace the entire thing, not just that section.

Then you may just want to wait till a new index is created (ie tomorrow).


(Phill Pafford) #11

ok still stumbling around, can you point me to the docs? maybe a working example as well?


(Mark Walkom) #12

The idea is to replace the entire existing template;

  1. GET _template/template_name
  2. Update the template and add your section
  3. POST _template/template_name

(Phill Pafford) #13

ok I ran

GET _template/logstash

and then with the JSON and added geoip section

POST _template/logstash
{
    ... Same JSON from get but with added section for GEOIP at the same level as the apache section ...
}

example

"nginx": {
},
"geoip": {
  "properties": {
    "country_iso_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "location": {
      "type": "geo_point"
    },
    "region_iso_code": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "continent_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "city_name": {
      "type": "keyword",
      "ignore_above": 1024
    },
    "region_name": {
      "type": "keyword",
      "ignore_above": 1024
    }
  }
},
"apache2": {
}

but I get this error

{
  "error": {
    "root_cause": [
      {
        "type": "action_request_validation_exception",
        "reason": "Validation Failed: 1: index patterns are missing;"
      }
    ],
    "type": "action_request_validation_exception",
    "reason": "Validation Failed: 1: index patterns are missing;"
  },
  "status": 400
}

(Mark Walkom) #14

I'd have to see the entire thing you are posting sorry.


(Phill Pafford) #15

solved this here Private ip geoip from dictionary