Template mapping dosen't work


(Ahmed HADDAD) #1

I'm trying to convert latitude and longitude to geo_point location in order to use it a map visulaztion.

this is my configuration:

input{

   file{
	path=>"C:\Users\Ahmed\Desktop\311_Service_Requests_from_2015.csv"
            start_position=>"beginning"
             sincedb_path=>NUL
       }
     }
filter{
    
      csv{
          separator=>","
          columns=>["Unique Key","Created Date","Closed Date","Agency","Agency Name","Complaint Type","Descriptor","Location Type","Incident Zip","Incident Address","Street Name","Cross Street 1","Cross Street 2","Intersection Street 1","Intersection Street 2","Address Type","City","Landmark","Facility Type","Status","Due Date","	Resolution Description","Resolution Action Updated Date","Community Board","Borough","X Coordinate (State Plane)","Y Coordinate (State Plane)","Park Facility Name","Park Borough","School Name","School Number","School Region","School Code","School Phone Number",	"School Address","School City","School State","School Zip","School Not Found","School or Citywide Complaint","Vehicle Type","Taxi Company Borough","Taxi Pick Up Location","Bridge Highway Name","Bridge Highway Direction","Road Ramp","Bridge Highway Segment","Garage Lot Name","Ferry Direction","Ferry Terminal Name","Latitude","Longitude","Location"]

         }
     mutate{
         convert=>["Latitude","float"]
	 convert=>["Longitude","float"]	
	 rename=>["Latitude","[location][lat]"]	
 	 rename=>["Longitude","[location][lon]"]
 	 }	            

     date{ 
    	match=>["Created Date","MM/dd/yyyy HH:mm:ss aa"]
    	target=>"Date"	
    	}
}

output{	
     
 elasticsearch {
         hosts=> "localhost"
         index=> "nyc311calls7"
        document_type=>"calls"
       }        
 stdout{codec=>rubydebug}
  }

i see in a youtube that i need t o add this template before creating the index pattern:

put _template/nyc*311*7
{
 "order":0,
  "index_patterns":"nyc*311*7",
  "mapping":{
   "_default":{
  "properties":{
    "location ":{
      "type":"geo_point"
       }
     }
   }
 }
}

but that dosen't convert the location to a geo_point, what i am doing wrong ?


(Bhavya R M) #2

@thomasneirynck ?

Thanks,
Bhavya


(Thomas Neirynck) #3

hi @akml_kk

What data type is your location field? Can you paste the mapping of your index here? Can you also paste a few example documents so we can see how are they getting indexed instead.


(Ahmed HADDAD) #4

hello,
i didn't get a location field in kibana,instead i get this two fields:
location.lat and location.lon and they are both floats.kibana

this is what i get for the mapping :
"mappings": {
"calls": {
"properties": {
"\tResolution Description": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Address Type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Agency": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Agency Name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Borough": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},

i can't paste it all, your site limit me to paste 7000 character

since i don't know how to get example documents what do you mean or how ?


(Ahmed HADDAD) #5

i can tell you also that there is no geoip field in mapping and this is how it looks the location field :

  "location": {
        "properties": {
          "lat": {
            "type": "float"
          },
          "lon": {
            "type": "float"
          }
        }

(Ahmed HADDAD) #6

this is the mapping of another index (just a simple example to show you how the mapping works in my machine) :

{
"hello_world": {
"mappings": {
"hello_world": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"path": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}


(Thomas Neirynck) #7

it seems like your field is not a geo_point, just a nested field with two floats.

It seems like your logstash script isn't picking up the template. I'm not a huge logstash expert, so I would ask this question in the forum there: https://discuss.elastic.co/c/logstash. Maybe you started indexing the data before creating the template?

With example document, I just meant an example of one of the docs, e.g. the individual incidents you see in the Discovery-app of Kibana. But at this point, it looks like mapping of the index is wrong.


(Ahmed HADDAD) #8

no , i created the template before ingesting the data , this is what logstash return before beginning the pipline :

ok,i will ask it then in logstash


(Magnus Bäck) #9

The template that Logstash installs for you applies to indexes whose name matches logstash-* but you're sending the documents to the nyc311calls7 index.


(Ahmed HADDAD) #10

but i have defined the template like this :

put _template/nyc*311*7
 {
 "order":0,
 "index_patterns":"nyc*311*7",
  "mapping":{
  "_default":{
  "properties":{
     "location ":{
      "type":"geo_point"
       }
 }
  }
}
}

(Magnus Bäck) #11

Okay, it wasn't clear that you created your template outside of Logstash.

"location ":{

Do you really have a space after "location"?

To debug things like this, create an index yourself after applying the template and fetch the mappings afterwards. Are they what you expect?


#12

Copying the solution from the other thread to this one... "mapping" should be "mappings", "location " needs that space to be removed. "_default" should be "_default_" (or "calls", since that is the document type).

This works.

PUT _template/nyc 
 {
    "order":0,
    "index_patterns":"nyc",
    "mappings":{
      "_default_":{
        "properties":{
          "location":{
            "type":"geo_point"
          }
        }
      }
    }
 }

(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.