Geo_point error in visualization


#1

Hi,

Using ELK stack 6.2.0

For weblogs getting -

Index pattern does not contain any of the following field types: geo_point

error in Kibana while creating the Visualize -> Map -> Coordinate Map

This is how logstash config file looks -

input {

   file {
     type => "weblog"
     path => "C:/elklogs/access_log.txt"
     start_position => "beginning"
     ignore_older => 0
   }
}

filter {

    if [type] == "weblog" {

        grok {
           patterns_dir => ".\patterns"
           match => { "message" => ["%{IBMACCESSLOG2}", "%{IBMACCESSLOG3}"]  }
        }
        date {
          match => [ "timestamp", "[dd/MMM/yyyy:HH:mm:ss Z]" ]
        }
        geoip {
            source => "clientip"
			target => "geoip"
        }
        useragent {
            source => "weblog_agent"
        }
        if "_grokparsefailure" not in [tags] {
            mutate {
                 remove_field => ["message"]
            }
        }		
    }

}

output {
   
     elasticsearch {
        action => "index"
        hosts => "localhost:9200"
        index => "sunview-%{+YYYY.MM}"
     }
     stdout {
        codec => rubydebug
     }	
}

logstash's rubydebug does show all the fields of geoip

Similarly, the index pattern fields screen from Kibana shows geoip fields minus geo_point

Please help on how to get geo_point correctly.

Thank you!


(Mark Walkom) #2

It's because it's geoip.location.lat and geoip.location.lon, when it should be an array of just two numbers under geoip.location.

Is that the only config you had?


#3

Thanks for looking.

There are other logs/config but none of them have IP data in it. Will that impact weblogs?

This is how the template data looks (using the default) -

GET /_template/logstash

{
  "logstash": {
    "order": 0,
    "version": 60001,
    "index_patterns": [
      "logstash-*"
    ],
    "settings": {
      "index": {
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "type": "text",
                "norms": false
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "keyword"
          },
          "geoip": {
            "dynamic": true,
            "properties": {
              "ip": {
                "type": "ip"
              },
              "location": {
                "type": "geo_point"
              },
              "latitude": {
                "type": "half_float"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          }
        }
      }
    },
    "aliases": {}
  }
}

(Mark Walkom) #4

Is that the default one?


#5

The config is not using any template so I believe that's the default one. Logstash logs showing logstash installing the template in ES -

[2018-02-10T19:10:49,193][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>nil}
[2018-02-10T19:10:49,202][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-02-10T19:10:49,222][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-02-10T19:10:49,249][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2018-02-10T19:10:49,302][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2018-02-10T19:10:49,472][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-02-10T19:10:49,968][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"C:/tools/ELK620/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"}

(Krunal Kalaria) #6

Hi @ggajanan,

Try This Template it may be worked.

After Change the template delete the old Index

Save this template in your logstash bin folder and give the path in your logstash config file like

template => "/usr/share/logstash/bin/template.json"
template_name => "test-*"

template.json
{
"template" : "test-",
"version" : 50001,
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true, "omit_norms" : false},
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"omit_norms" : false
}
}
}, {
"string_fields" : {
"match" : "
",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "omit_norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date", "include_in_all": false },
"@version": { "type": "keyword", "include_in_all": false },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
},
"location": { "type": "geo_point" }
}
}
}
}

Thanks & Reagrds,
Krunal.


#7

Hi Krunal,

I had to make some changes to install it in 6.2 and installed it in ES using Kibana console -

PUT _template/weblog.template
{
  "index_patterns": ["weblog*"],
  "settings": {
    "number_of_shards": 1,
    "index.refresh_interval": "5s"
  },
	"mappings": {
		"default": {
			"dynamic_templates": [
				{
					"message_field": {
						"path_match": "message",
						"match_mapping_type": "string",
						"mapping": {
							"type": "text",
							"omit_norms": false
						}
					}
				},
				{
					"string_fields": {
						"match": "",
						"match_mapping_type": "string",
						"mapping": {
							"type": "text",
							"omit_norms": false,
							"fields": {
								"keyword": {
									"type": "keyword",
									"ignore_above": 256
								}
							}
						}
					}
				}
			],
			"properties": {
				"@timestamp": {
					"type": "date"
				},
				"@version": {
					"type": "keyword"
				},
				"geoip": {
					"dynamic": true,
					"properties": {
						"ip": {
							"type": "ip"
						},
						"location": {
							"type": "geo_point"
						},
						"latitude": {
							"type": "half_float"
						},
						"longitude": {
							"type": "half_float"
						}
					}
				},
				"location": {
					"type": "geo_point"
				}
			}
		}
	}
}

Modified o/p of logstash config to make use of this template -

output {
   
     elasticsearch {
        action => "index"
        hosts => "localhost:9200"
        index => "weblog-%{+YYYY.MM}"
		template_name => "weblog.template"
     }
     stdout {
        codec => rubydebug
     }	
}

Now hitting following error -

[2018-02-12T18:23:04,534][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x76158012 sleep>"}
[2018-02-12T18:23:04,561][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-02-12T18:23:06,113][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"weblog-2018.02", :_type=>"doc", :_routi
ng=>nil}, #<LogStash::Event:0x770eb729>], :response=>{"index"=>{"_index"=>"weblog-2018.02", "_type"=>"doc", "_id"=>"1R5UjGEBVbzVFun6OmAG", "status"=>400, "error"=>{"type"=>"illegal_argument_exception"
, "reason"=>"[geoip.location] is defined as an object in mapping [doc] but this name is already used for a field in other types"}}}}
[2018-02-12T18:23:06,115][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"weblog-2018.02", :_type=>"doc", :_routi
ng=>nil}, #<LogStash::Event:0x74961af6>], :response=>{"index"=>{"_index"=>"weblog-2018.02", "_type"=>"doc", "_id"=>"4R5UjGEBVbzVFun6OmAH", "status"=>400, "error"=>{"type"=>"illegal_argument_exception"
, "reason"=>"[geoip.location] is defined as an object in mapping [doc] but this name is already used for a field in other types"}}}}
[2018-02-12T18:23:06,114][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"weblog-2018.02", :_type=>"doc", :_routi
ng=>nil}, #<LogStash::Event:0x3def49fe>], :response=>{"index"=>{"_index"=>"weblog-2018.02", "_type"=>"doc", "_id"=>"yR5UjGEBVbzVFun6OmAF", "status"=>400, "error"=>{"type"=>"illegal_argument_exception"
, "reason"=>"[geoip.location] is defined as an object in mapping [doc] but this name is already used for a field in other types"}}}}

This used to work perfectly on ELK stack 5.3.0 for. Not sure what's wrong here and stuck on this. Any help is really appreciated.

Thank you!


#8

@warkolmHi, can you please look into this?

Thanks


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.