Visualization of lat lon coordinates on Kibana's graph

citanionarrei · November 2, 2017, 11:13am

hi
i have a problem
i have log files with two fields: "lat" and "lon"

and this error on kibana visualization

"the "meetup-*" index pattern does not contain any of the following field types: geo_point"

how can i resolve this problem?
how i can set logstash for it
Now i'm using this conf. on logstash:

input {file {
path => ["C:\Users\Lock\Desktop\meetup.json"]  
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
type => "log"
}}

filter
{
if [type] == "log"
{
grok {	
	match => ["message", "(?<meetup>{\".*\".*})"]
	break_on_match => true
	add_field => { "type" => "json_meetup"} 	
	}}
if "_grokparsefailure" in [tags]{drop {}}	
else {json {source => "meetup"}	}
}

output{ elasticsearch {
index => "meetup-%{+YYYY.MM.dd}"
document_type => "json_meetup"
codec => json_lines
}

thanks

mikelD · November 2, 2017, 3:29pm

You have to define the field that contains the latitude and longitude as a geo_point, in the index mapping, for reference in this link https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html you have the different ways it can be specified.

You can define the field in a index template for the new indexes.

citanionarrei · November 2, 2017, 3:33pm

Thank you for your answer...
I am not yet very experienced
and I did not understand how I can do this.
Can you help me?
where do I build the template? on logstash or on dev tool on kibana?

this is my json

i'm using this as source of logs
http://stream.meetup.com/2/rsvps

how can i create template for logstash and create field for geoip?
this dont work

filter{
grok {
match => ["message", "(?{".".})"]
break_on_match => true
add_field => { "type" => "json_meetup"}
}

mutate {
add_field => { "[venue][lat]" => "%{venue.lat}" }
add_field => { "[venue][lon]" => "%{venue.lon}" }
}
mutate {
convert => {"[venue][lat]" => "float"}
convert => {"[venue][lon]" => "float"}
}

mikelD · November 3, 2017, 9:15am

First, you need to create a template for your index that has a field mapped as geo_point. You can do this with the Dev Tool on kibana,with this tool you can communicate with the elasticsearch node and make searches, insert data or configure parameters.

You can see the mapping of one of your index with this command:
GET [index_id]/_mapping
in my case
GET gps-2017.08.10/_mapping

Using the mapping as a template, you can add a field(for example location) and map it as a geo_point (It's better to add a new field for the coordinates because if you use one you already had mapped differently in previous index, elasticsearch will give you error becouse the same field is mapped differently in two related indexes):

"gps-2017.08.10": {
"mappings": {
"logs": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"coordinates": {
"type": "geo_point"
},
"dev_addr": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dev_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
"host": {
"type": "ip"
}
}
}
}

Then you can use this mapping to create a template, like this one:

{
"template": "gps-*",
"order": 0,
"settings": {
"index.mapping.ignore_malformed": true
},
"mappings": {
"logs": {
"properties": {
"coordinates": {
"type": "geo_point"
},
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
"dev_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
"host": {
"type": "ip"
},
"dev_addr": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dev_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}

To add the new template to elasticsearch, you insert the next command in the dev tool followed by the template, the index identifier in my example would be 'gps':
PUT _template/[index_identifier]
[Template]

Once you have a template with a geo_point field defined, you can insert data in this index, the field coordinates must be formatted like this:

Geo-point as an object

"coordinates": {
"lat": 41.12,
"lon": -71.34
}

Geo-point as a string

"coordinates": "41.12,-71.34"

Geo-point as a geohash

"coordinates": "drm3btev3e86"

Geo-point as an array

"coordinates": [ -71.34, 41.12 ]

If you do this right and you have no conflict in the indexes mapping the maptiles pluging from kibana should work correctly.

citanionarrei · November 3, 2017, 9:44am

ohhhh many thanks friend..
ok i now i see it on my mapping

 "venue": {
        "properties": {
          "lat": {
            "type": "float"
          },
          "lon": {
            "type": "float"
          },

than ... i create an external template "meetup1.json" that i can load with output of logstash in this way and overwrite the defoult template
output

{ 
elasticsearch {
index => "meetup"
document_type => "json_meetup"
manage_template => true
template => "C:\Users\Lock\Desktop\meetup1.json"
template_overwrite => "true"
codec => json_lines
}

Than if i change " float " with "geo_point" can i resolve the problem on kibana?

must i add location field ?

mikelD · November 3, 2017, 10:15am

you need to modify the lat and lon so they are inside another field wich is defined as geo_point, because if you don't do it, elastic won't identify them as coordinates just numbers.

The data must have a format like this:

"venue":{
"venue_name":"Name",
...
"coordinates":{
"lat":-42.7,
"long":-25.1
},
...
}

You will need to parse your data source because with the data you are receiving you don't have an easy way to do it.

it's not necessary to add the location field on logstash, because you can format it to geo_point, Logstash only knows standard data types. It is recommended to have the template in my experience to avoid problems with conflicting mappings in the indexes.

It happened to me that i was sending a data of type float but because the value of the field was exact (24.0) it mapped it as a long in the index, this kind of things creates a lot of problems.

citanionarrei · November 4, 2017, 8:30am

if my "coordinates" are formatted like these ways, can i use only the mapping "coordinates": {type:geo_point} ?

citanionarrei · November 4, 2017, 9:31am

ok now i have

fields"venue.lat" and " venue.lon" are now type: geo_point

my template is

PUT _template/template_1
{
 "template":"meetup-*",
 "settings":{
"number_of_shards": 1
},

"mappings": {
  "json_meetup": {
    "properties": {
      "venue": {
        "properties": {
          "lat": {
            "type": "geo_point"
          },
          "lon": {
            "type": "geo_point"
          }
          
          
        } } }

but when i try to visualize my points i can choose only venue.lat OR venue.lon field than in this way it is impossible a visualization

mikelD · November 6, 2017, 7:27am

That happens because you have defined both venue.lat and venue.long as a geo_point, your geo_point must be a unique field that contains the lat and lon.

imagen

In this Image the field "location" is defined as a geo_point, and the fields "lat" and "lon" are two subfields of location.

citanionarrei · November 6, 2017, 2:48pm

how can i create it?
i'm using this on logstash

mutate {
 add_field => { "[geopoint][latitude]" => "%{[venue][lat]}" }
 add_field => { "[geopoint][longitude]" => "%{[venue][lon]}" }
 remove_field =>["meetup","message","group","event"]	}	   
			   
mutate {
 convert => {
   "[geopoint][latitude]" => "float"
   "[geopoint][longitude]" => "float"
}

i obtain in my json this:

Screenshot (54)

and on kibana

Screenshot (55)

mapping that i'm using

{
 "template":"meetup-*",
"settings":{
"number_of_shards": 1
},

 "mappings": {
  "json_meetup": {
    "properties": {
	  "geopoint": {
	  "type": "geo_point",}
        
}}}}

maybe is it broken?

mikelD · November 6, 2017, 3:05pm

In the template after "type": "geo_point" you have a , that is a syntax mistake and shouldn't allow you to load the template, try using the command GET _templating/[name of the template] to check that it has been loaded correctly , the name you have used with the command PUT.

citanionarrei · November 6, 2017, 5:13pm

i have problem

my json is:

{
 "venue": {
  "venue_name": "Facebook",
  "lon": -6.23911,
  "lat": 53.343712,
  "venue_id": 23843832
},
"visibility": "public",
"rsvp_id": 1697704652,
"geopoint": {
  "latitude": 53.343712,
  "longitude": -6.23911
},
"type": ["log","json_meetup"],
"mtime": 1509981678435,
"path": "C:\\Users\\Lock\\Desktop\\meetup.json",
"@timestamp": "2017-11-06T16:49:04.147Z",
"response": "yes",
"@version": "1",
"host": "Lock",
"guests": 0,
"member": {
  "member_id": 202905630,
  "member_name": "Brian Quinlan"
    }
  },
 "fields": {
"@timestamp": [1509986944147]
}
}

my configuration file of logstash is this

input 
{file 
{
path => ["C:\Users\Lock\Desktop\meetup.json"]  
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
type => "log"
}
}

filter{
grok {
match => ["message", "(?<meetup>{\".*\".*})"]
break_on_match => true
add_field => { "type" => "json_meetup"} 	
}
if "_grokparsefailure" in [tags]{drop {}}	
else {
json {source => "meetup"}}	 
mutate {
 add_field => { "[geopoint][latitude]" => "%{[venue][lat]}" }
 add_field => { "[geopoint][longitude]" => "%{[venue][lon]}" }
 remove_field =>["meetup","message","group","event"]	}	   				   
mutate {
 convert => {
   "[geopoint][latitude]" => "float"
   "[geopoint][longitude]" => "float"
}}}			   		   			   
		
output
{ elasticsearch {
index => "meetup-%{+YYYY.MM.dd}"
document_type => "json_meetup"

manage_template => true
template => "C:\Users\Lock\Desktop\template3.json"
template_overwrite => true
codec => json
}

stdout { codec => rubydebug }}

and my template that i load on elastic with logstash is this

{
  "template":"meetup-*",
"settings":{
"number_of_shards": 1
},

"mappings": {
  "json_meetup": {
    "properties": {
	"geopoint": {
		"properties": {
            "latitude": {
            "type": "geo_point"
          },
          "longitude": {
            "type": "geo_point"
          }
         }
       }
     }
   }
 }
 }

in this way mapping dont work

mikelD · November 7, 2017, 8:54am

Your template is wrong,it should be like this:

{
"template":"meetup-*",
"settings":{
"number_of_shards": 1
},

"mappings": {
"json_meetup": {
"properties": {
"geopoint": {
"type": "geo_point"
}
}
}
}
}

You should check the } there seems to be to much, check with a editor like atom to see if they sintaxis of the JSON is correct.

You don't have to define the latitude and longitude as geo_point but the field that contains them.

citanionarrei · November 7, 2017, 8:56am

i have also tried with it but doesn't work

citanionarrei · November 7, 2017, 2:42pm

I solved the problem thanks to you for the help..

im using this template

 {
    "template":"meetup-*",
"settings":{
"number_of_shards": 1
			},

"mappings": {
	"log": {
		"properties": {
			"location": {
				"type": "geo_point"
				}
			}
		}
	}
}

and this on logstash

mutate {

	 add_field => { "[location]" => "%{[venue][lat]},%{[venue][lon]}"}
   }

 kv {
	source => "location"
	target => "location"
	field_split => ","}  
   
 mutate {
	convert => { "[location][lat]" => "float" }
	convert => { "[location][lon]" => "float" }

I would have another curiosity: can I map for any field?
if I wanted to do it for example i want change the type of "guest" from "number "to "string" can i make in this way?

 {
    "template":"meetup-*",
"settings":{
"number_of_shards": 1
			},

"mappings": {
	"log": {
		"properties": {
			"location": {
				"type": "geo_point"
				}


             "guests":{
				"type": "text"
				}



			}
		}
	}
}

becouse so i have problems and template in this way dont work!

mikelD · November 7, 2017, 2:57pm

I'm not sure if it would work, but even if it worked you would have a problem because the string would be in ASCII and if it converts the number directly as ASCII code it will print other characters. For example: the character '1' in ASCII has the decimal number 49. And the decimal number 1, and many others are special characters like backspace and the like.

You will have to parse the field and convert it to text before sending it.

citanionarrei · November 7, 2017, 3:00pm

yes it was only an example..
i speaked for the template and if the structure for the mapping is correct.
it is the way to map all fields?

mikelD · November 8, 2017, 10:49am

Yes, you can map for nay kind of field, automatically elasticsearch detects the type of data and maps it to the index, but some special types like geo_point and ip must be mapped manually. In my experience, I recommend mapping all the fields in the template to avoid problems if it maps a field incorrectly.

About the structure you have to separate the diferent fiels with a ',', like this:

"batterylevel": {
"type": "long"
},
"longitude": {
"type": "float"
}

system · December 6, 2017, 10:50am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
"Getting started"-Coordinate Map - Tutorial brocken Kibana	6	449	August 16, 2018
Help needed in loading map using lat and lon value Kibana	3	557	November 30, 2017
Unable to load lat and lon values in map view Logstash	6	709	November 30, 2017
Latitude and Longitude Field conversion to Geo Point Kibana	3	4665	February 22, 2017
Cannot find geolocation in Kibana Visualization Kibana	12	3094	December 7, 2017

Visualization of lat lon coordinates on Kibana's graph

Related topics