Template mapping dosen't work

I'm trying to convert latitude and longitude to geo_point location in order to use it a map visulaztion.

this is my configuration:

input{

   file{
	path=>"C:\Users\Ahmed\Desktop\311_Service_Requests_from_2015.csv"
            start_position=>"beginning"
             sincedb_path=>NUL
       }
     }
filter{
    
      csv{
          separator=>","
          columns=>["Unique Key","Created Date","Closed Date","Agency","Agency Name","Complaint Type","Descriptor","Location Type","Incident Zip","Incident Address","Street Name","Cross Street 1","Cross Street 2","Intersection Street 1","Intersection Street 2","Address Type","City","Landmark","Facility Type","Status","Due Date","	Resolution Description","Resolution Action Updated Date","Community Board","Borough","X Coordinate (State Plane)","Y Coordinate (State Plane)","Park Facility Name","Park Borough","School Name","School Number","School Region","School Code","School Phone Number",	"School Address","School City","School State","School Zip","School Not Found","School or Citywide Complaint","Vehicle Type","Taxi Company Borough","Taxi Pick Up Location","Bridge Highway Name","Bridge Highway Direction","Road Ramp","Bridge Highway Segment","Garage Lot Name","Ferry Direction","Ferry Terminal Name","Latitude","Longitude","Location"]

         }
     mutate{
         convert=>["Latitude","float"]
	 convert=>["Longitude","float"]	
	 rename=>["Latitude","[location][lat]"]	
 	 rename=>["Longitude","[location][lon]"]
 	 }	            

     date{ 
    	match=>["Created Date","MM/dd/yyyy HH:mm:ss aa"]
    	target=>"Date"	
    	}
}

output{	
     
 elasticsearch {
         hosts=> "localhost"
         index=> "nyc311calls7"
        document_type=>"calls"
       }        
 stdout{codec=>rubydebug}
  }

i see in a youtube that i need t o add this template before creating the index pattern:

put _template/nyc*311*7
{
 "order":0,
  "index_patterns":"nyc*311*7",
  "mapping":{
   "_default":{
  "properties":{
    "location ":{
      "type":"geo_point"
       }
     }
   }
 }
}

but that dosen't convert the location to a geo_point, what i am doing wrong ?

@thomasneirynck ?

Thanks,
Bhavya

hi @akml_kk

What data type is your location field? Can you paste the mapping of your index here? Can you also paste a few example documents so we can see how are they getting indexed instead.

hello,
i didn't get a location field in kibana,instead i get this two fields:
location.lat and location.lon and they are both floats.kibana

this is what i get for the mapping :
"mappings": {
"calls": {
"properties": {
"\tResolution Description": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Address Type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Agency": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Agency Name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Borough": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},

i can't paste it all, your site limit me to paste 7000 character

since i don't know how to get example documents what do you mean or how ?

i can tell you also that there is no geoip field in mapping and this is how it looks the location field :

  "location": {
        "properties": {
          "lat": {
            "type": "float"
          },
          "lon": {
            "type": "float"
          }
        }

this is the mapping of another index (just a simple example to show you how the mapping works in my machine) :

{
"hello_world": {
"mappings": {
"hello_world": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}

it seems like your field is not a geo_point, just a nested field with two floats.

It seems like your logstash script isn't picking up the template. I'm not a huge logstash expert, so I would ask this question in the forum there: https://discuss.elastic.co/c/logstash. Maybe you started indexing the data before creating the template?

With example document, I just meant an example of one of the docs, e.g. the individual incidents you see in the Discovery-app of Kibana. But at this point, it looks like mapping of the index is wrong.

no , i created the template before ingesting the data , this is what logstash return before beginning the pipline :

ok,i will ask it then in logstash

The template that Logstash installs for you applies to indexes whose name matches logstash-* but you're sending the documents to the nyc311calls7 index.

but i have defined the template like this :

put _template/nyc*311*7
 {
 "order":0,
 "index_patterns":"nyc*311*7",
  "mapping":{
  "_default":{
  "properties":{
     "location ":{
      "type":"geo_point"
       }
 }
  }
}
}

Okay, it wasn't clear that you created your template outside of Logstash.

"location ":{

Do you really have a space after "location"?

To debug things like this, create an index yourself after applying the template and fetch the mappings afterwards. Are they what you expect?

1 Like

Copying the solution from the other thread to this one... "mapping" should be "mappings", "location " needs that space to be removed. "_default" should be "_default_" (or "calls", since that is the document type).

This works.

PUT _template/nyc 
 {
    "order":0,
    "index_patterns":"nyc",
    "mappings":{
      "_default_":{
        "properties":{
          "location":{
            "type":"geo_point"
          }
        }
      }
    }
 }
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.