Visualization of lat lon coordinates on Kibana's graph

hi
i have a problem
i have log files with two fields: "lat" and "lon"


and this error on kibana visualization

"the "meetup-*" index pattern does not contain any of the following field types: geo_point"

how can i resolve this problem?
how i can set logstash for it
Now i'm using this conf. on logstash:

input {file {
path => ["C:\Users\Lock\Desktop\meetup.json"]  
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
type => "log"
}}

filter
{
if [type] == "log"
{
grok {	
	match => ["message", "(?<meetup>{\".*\".*})"]
	break_on_match => true
	add_field => { "type" => "json_meetup"} 	
	}}
if "_grokparsefailure" in [tags]{drop {}}	
else {json {source => "meetup"}	}
}

output{ elasticsearch {
index => "meetup-%{+YYYY.MM.dd}"
document_type => "json_meetup"
codec => json_lines
}

thanks

You have to define the field that contains the latitude and longitude as a geo_point, in the index mapping, for reference in this link https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html you have the different ways it can be specified.

You can define the field in a index template for the new indexes.

Thank you for your answer...
I am not yet very experienced
and I did not understand how I can do this.
Can you help me?
where do I build the template? on logstash or on dev tool on kibana?

this is my json

i'm using this as source of logs
http://stream.meetup.com/2/rsvps

how can i create template for logstash and create field for geoip?
this dont work

filter{
grok {
match => ["message", "(?{".".})"]
break_on_match => true
add_field => { "type" => "json_meetup"}
}

mutate {
add_field => { "[venue][lat]" => "%{venue.lat}" }
add_field => { "[venue][lon]" => "%{venue.lon}" }
}
mutate {
convert => {"[venue][lat]" => "float"}
convert => {"[venue][lon]" => "float"}
}

First, you need to create a template for your index that has a field mapped as geo_point. You can do this with the Dev Tool on kibana,with this tool you can communicate with the elasticsearch node and make searches, insert data or configure parameters.

You can see the mapping of one of your index with this command:
GET [index_id]/_mapping
in my case
GET gps-2017.08.10/_mapping

Using the mapping as a template, you can add a field(for example location) and map it as a geo_point (It's better to add a new field for the coordinates because if you use one you already had mapped differently in previous index, elasticsearch will give you error becouse the same field is mapped differently in two related indexes):

"gps-2017.08.10": {
"mappings": {
"logs": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"coordinates": {
"type": "geo_point"
},
"dev_addr": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dev_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
"host": {
"type": "ip"
}
}
}
}

Then you can use this mapping to create a template, like this one:

{
"template": "gps-*",
"order": 0,
"settings": {
"index.mapping.ignore_malformed": true
},
"mappings": {
"logs": {
"properties": {
"coordinates": {
"type": "geo_point"
},
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
"dev_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
"host": {
"type": "ip"
},
"dev_addr": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"dev_id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}

To add the new template to elasticsearch, you insert the next command in the dev tool followed by the template, the index identifier in my example would be 'gps':
PUT _template/[index_identifier]
[Template]

Once you have a template with a geo_point field defined, you can insert data in this index, the field coordinates must be formatted like this:

  • Geo-point as an object

"coordinates": {
"lat": 41.12,
"lon": -71.34
}

  • Geo-point as a string

"coordinates": "41.12,-71.34"

  • Geo-point as a geohash

"coordinates": "drm3btev3e86"

  • Geo-point as an array

"coordinates": [ -71.34, 41.12 ]

If you do this right and you have no conflict in the indexes mapping the maptiles pluging from kibana should work correctly.

1 Like

ohhhh many thanks friend..
ok i now i see it on my mapping

 "venue": {
        "properties": {
          "lat": {
            "type": "float"
          },
          "lon": {
            "type": "float"
          },

than ... i create an external template "meetup1.json" that i can load with output of logstash in this way and overwrite the defoult template
output

{ 
elasticsearch {
index => "meetup"
document_type => "json_meetup"
manage_template => true
template => "C:\Users\Lock\Desktop\meetup1.json"
template_overwrite => "true"
codec => json_lines
}

Than if i change " float " with "geo_point" can i resolve the problem on kibana?

must i add location field ?

you need to modify the lat and lon so they are inside another field wich is defined as geo_point, because if you don't do it, elastic won't identify them as coordinates just numbers.

The data must have a format like this:

"venue":{
"venue_name":"Name",
...
"coordinates":{
"lat":-42.7,
"long":-25.1
},
...
}

You will need to parse your data source because with the data you are receiving you don't have an easy way to do it.

it's not necessary to add the location field on logstash, because you can format it to geo_point, Logstash only knows standard data types. It is recommended to have the template in my experience to avoid problems with conflicting mappings in the indexes.

It happened to me that i was sending a data of type float but because the value of the field was exact (24.0) it mapped it as a long in the index, this kind of things creates a lot of problems.

if my "coordinates" are formatted like these ways, can i use only the mapping "coordinates": {type:geo_point} ?

ok now i have

fields"venue.lat" and " venue.lon" are now type: geo_point

my template is

PUT _template/template_1
{
 "template":"meetup-*",
 "settings":{
"number_of_shards": 1
},

"mappings": {
  "json_meetup": {
    "properties": {
      "venue": {
        "properties": {
          "lat": {
            "type": "geo_point"
          },
          "lon": {
            "type": "geo_point"
          }
          
          
        } } }

but when i try to visualize my points i can choose only venue.lat OR venue.lon field than in this way it is impossible a visualization

That happens because you have defined both venue.lat and venue.long as a geo_point, your geo_point must be a unique field that contains the lat and lon.

imagen

In this Image the field "location" is defined as a geo_point, and the fields "lat" and "lon" are two subfields of location.

how can i create it?
i'm using this on logstash

mutate {
 add_field => { "[geopoint][latitude]" => "%{[venue][lat]}" }
 add_field => { "[geopoint][longitude]" => "%{[venue][lon]}" }
 remove_field =>["meetup","message","group","event"]	}	   
			   
mutate {
 convert => {
   "[geopoint][latitude]" => "float"
   "[geopoint][longitude]" => "float"
}

i obtain in my json this:

Screenshot (54)

and on kibana

Screenshot (55)

mapping that i'm using

{
 "template":"meetup-*",
"settings":{
"number_of_shards": 1
},

 "mappings": {
  "json_meetup": {
    "properties": {
	  "geopoint": {
	  "type": "geo_point",}
        
}}}}

maybe is it broken?

In the template after "type": "geo_point" you have a , that is a syntax mistake and shouldn't allow you to load the template, try using the command GET _templating/[name of the template] to check that it has been loaded correctly , the name you have used with the command PUT.

i have problem :disappointed_relieved:

my json is:

{
 "venue": {
  "venue_name": "Facebook",
  "lon": -6.23911,
  "lat": 53.343712,
  "venue_id": 23843832
},
"visibility": "public",
"rsvp_id": 1697704652,
"geopoint": {
  "latitude": 53.343712,
  "longitude": -6.23911
},
"type": ["log","json_meetup"],
"mtime": 1509981678435,
"path": "C:\\Users\\Lock\\Desktop\\meetup.json",
"@timestamp": "2017-11-06T16:49:04.147Z",
"response": "yes",
"@version": "1",
"host": "Lock",
"guests": 0,
"member": {
  "member_id": 202905630,
  "member_name": "Brian Quinlan"
    }
  },
 "fields": {
"@timestamp": [1509986944147]
}
}

my configuration file of logstash is this

input 
{file 
{
path => ["C:\Users\Lock\Desktop\meetup.json"]  
start_position => "beginning"
sincedb_path => "/dev/null"
exclude => "*.gz"
type => "log"
}
}

filter{
grok {
match => ["message", "(?<meetup>{\".*\".*})"]
break_on_match => true
add_field => { "type" => "json_meetup"} 	
}
if "_grokparsefailure" in [tags]{drop {}}	
else {
json {source => "meetup"}}	 
mutate {
 add_field => { "[geopoint][latitude]" => "%{[venue][lat]}" }
 add_field => { "[geopoint][longitude]" => "%{[venue][lon]}" }
 remove_field =>["meetup","message","group","event"]	}	   				   
mutate {
 convert => {
   "[geopoint][latitude]" => "float"
   "[geopoint][longitude]" => "float"
}}}			   		   			   
		
output
{ elasticsearch {
index => "meetup-%{+YYYY.MM.dd}"
document_type => "json_meetup"

manage_template => true
template => "C:\Users\Lock\Desktop\template3.json"
template_overwrite => true
codec => json
}

stdout { codec => rubydebug }}

and my template that i load on elastic with logstash is this

{
  "template":"meetup-*",
"settings":{
"number_of_shards": 1
},

"mappings": {
  "json_meetup": {
    "properties": {
	"geopoint": {
		"properties": {
            "latitude": {
            "type": "geo_point"
          },
          "longitude": {
            "type": "geo_point"
          }
         }
       }
     }
   }
 }
 }

in this way mapping dont work

Your template is wrong,it should be like this:

{
"template":"meetup-*",
"settings":{
"number_of_shards": 1
},

"mappings": {
"json_meetup": {
"properties": {
"geopoint": {
"type": "geo_point"
}
}
}
}
}

You should check the } there seems to be to much, check with a editor like atom to see if they sintaxis of the JSON is correct.

You don't have to define the latitude and longitude as geo_point but the field that contains them.

i have also tried with it but doesn't work :face_with_raised_eyebrow:

I solved the problem thanks to you for the help..

im using this template

 {
    "template":"meetup-*",
"settings":{
"number_of_shards": 1
			},

"mappings": {
	"log": {
		"properties": {
			"location": {
				"type": "geo_point"
				}
			}
		}
	}
}

and this on logstash

mutate {

	 add_field => { "[location]" => "%{[venue][lat]},%{[venue][lon]}"}
   }

 kv {
	source => "location"
	target => "location"
	field_split => ","}  
   
 mutate {
	convert => { "[location][lat]" => "float" }
	convert => { "[location][lon]" => "float" }

I would have another curiosity: can I map for any field?
if I wanted to do it for example i want change the type of "guest" from "number "to "string" can i make in this way?

 {
    "template":"meetup-*",
"settings":{
"number_of_shards": 1
			},

"mappings": {
	"log": {
		"properties": {
			"location": {
				"type": "geo_point"
				}


             "guests":{
				"type": "text"
				}



			}
		}
	}
}

becouse so i have problems and template in this way dont work!

I'm not sure if it would work, but even if it worked you would have a problem because the string would be in ASCII and if it converts the number directly as ASCII code it will print other characters. For example: the character '1' in ASCII has the decimal number 49. And the decimal number 1, and many others are special characters like backspace and the like.

You will have to parse the field and convert it to text before sending it.

yes it was only an example..
i speaked for the template and if the structure for the mapping is correct.
it is the way to map all fields?

Yes, you can map for nay kind of field, automatically elasticsearch detects the type of data and maps it to the index, but some special types like geo_point and ip must be mapped manually. In my experience, I recommend mapping all the fields in the template to avoid problems if it maps a field incorrectly.

About the structure you have to separate the diferent fiels with a ',', like this:

"batterylevel": {
"type": "long"
},
"longitude": {
"type": "float"
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.