Geoip index mapping defaults to using logstash-*


(Christopher Pax) #1

It seemes that when useing the geoip filter. It creates a mapping to convert the location to a geo_point. But only for index's called logstash-*. But I named my index's apache-*. I have been trying to utilize the "template" attribute of the elasticsearch output plugin to do this. but it seems, the only way to get the tile map to work with my custom index name is to create my own template using the API.

my logstash configuration:

input {
	# logs provided by rsyslog from other servers
	syslog { # apache logs
	        host => "0.0.0.0"
	        port => 5544
	        type => "apache"
	}
	syslog { # kernel logs
	        host => "0.0.0.0"
	        port => 5548
	        type => mssg
	        use_labels => true
	}
}

filter {
	if [type] == "apache" {
	        grok {  
	                match => [ "message", "%{COMBINEDAPACHELOG}" ]
	        }
	        geoip { 
	                source => "clientip"
	                target => "geoip"
	                database => "/etc/logstash/GeoLiteCity.dat"
	                add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
	                add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]

	        }
	        mutate {
	                convert => [ "[geoip][coordinates]", "float"]
	        }

	}

	if [type] == "mssg" {
	        grok {  
	                match => { "message" => "%{SYSLOGLINE}" }

	        }
	}
}



output {

	if [type] == "apache" {
	        elasticsearch {
	                hosts => ["elasticserver"]
	                index => "apache-%{+YYYY.MM.dd}"
			# was trying to use # template => /etc/elasticsearch/apache.json
			# but could not figure out what to put in apache.json to get this to work
	        }

	}
	if [type] == "mssg" {
	        elasticsearch {
	                hosts => ["elasticserver"]
	                index => "mssg-%{+YYYY.MM.dd}"
	        }
	}
}

Running a tile map on the apache-* index at this point would not work, and kibana would throw an error
saying ''no geo_point found''.

results of GET _template before creating my own:

{
  "logstash": {
    "order": 0,
    "template": "logstash-*",
    "settings": {
      "index": {
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "default": {
        "dynamic_templates": [
          {
            "message_field": {
              "mapping": {
                "fielddata": {
                  "format": "disabled"
                },
                "index": "analyzed",
                "omit_norms": true,
                "type": "string"
              },
              "match_mapping_type": "string",
              "match": "message"
            }
          },
          {
            "string_fields": {
              "mapping": {
                "fielddata": {
                  "format": "disabled"
                },
                "index": "analyzed",
                "omit_norms": true,
                "type": "string",
                "fields": {
                  "raw": {
                    "ignore_above": 256,
                    "index": "not_analyzed",
                    "type": "string"
                  }
                }
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "_all": {
          "omit_norms": true,
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "geoip": {
            "dynamic": true,
            "properties": {
              "ip": {
                "type": "ip"
              },
              "latitude": {
                "type": "float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "float"
              }
            }
          },
          "@version": {
            "index": "not_analyzed",
            "type": "string"
          }
        }
      }
    },
    "aliases": {
      
    }
  }
}

as you can see, the geo_point mapping is there. This was put in by default.

this is how I created my own template PUT /_template/apache-template:

{
    "order": 0,
    "template": "apache-*",
    "settings": {
      "index": {
	"number_of_shards": "1"
      }
    },
    "mappings": {
      "apache": {
	"properties": {
	  "geoip": {
	    "dynamic": true,
	    "type": "object",
	    "properties": {
	      "location": {
	        "type": "geo_point"
	      }
	    }
	  }
	}
      }
    },
    "aliases": {}

}

after this the tile map would work. I used this same code I have here for apache-template in the template => /path/to/template.json, but
that did not work.

  • is there a way to utilize the template attribute in the elasticsearch output to handle this mapping?
  • am I doing this wrong?

(Mark Walkom) #2

What is the actual config you used to refer to the template?


(Christopher Pax) #3

@warkolm

I tried a few variations, and cant recall what they were. The question is what should be the contents of this template file.


(Mark Walkom) #4

https://www.elastic.co/guide/en/elasticsearch/reference/2.3/dynamic-templates.html#dynamic-templates


(Christopher Pax) #5

Okay. But I am not sure how to apply it for just one property of geoip. also, there is nothing on that page that states "this is exactly the format you need for the 'template' attribute of logstash elasticsearch output"

it is unclear how to use that feature. I am hoping for more of a: "use this for your logstash configuration <insertcode here>, and use this for the contents of the template file <insertcode here>" as a response.


(Mark Walkom) #6

This is the LS 2.X template you can adapt to your own needs - https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template-es2x.json


(Joseph Spencer) #7

@warkolm I don't feel the following question (asked by @christopher_pax) has been answered:

is there a way to utilize the template attribute in the elasticsearch output to handle this mapping?

I'm facing this same issue.

@christopher_pax was able to get his mapping to work by invoking PUT /_template/apache-template, but it would be easier if the template attribute of the output plugin could do this instead.


(Mark Walkom) #8

It can, just refer to the file you created.


(system) #9