No geo_point fields logstash 6

Elastic Stack Logstash
1

No Compatible Fields: The "apache-access_*" index pattern does not contain any of the following field types: geo_point

Logstash geo config that worked fine for logstash 5.6

geoip {
  source => "clientip"
  target => "geoip"
  add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
  add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
}
mutate {
  convert => [ "[geoip][coordinates]", "float"]
}
2

bump

3

Do you use an index template that maps [geoip][coordinates] as geo_point?

4

Please see https://www.elastic.co/blog/geoip-in-the-elastic-stack for common issues with this.

5

This quote from the article leads me to believe that I don't have to worry about Elasticsearch, is that correct?

What this all means is that any field called geoip.location, which is how we refer to these nested fields, that is sent to Elasticsearch will be automatically mapped as a geo_point.

6

How do I check?

7

This quote from the article leads me to believe that I don't have to worry about Elasticsearch, is that correct?

No, because of

Assuming we are using the default Logstash or Filebeat index name pattern ...

at the beginning of the paragraph.

How do I check?

If you haven't uploaded such an index template yourself it's safe to assume you don't have one. You should either

  • not override the default index name in your elasticsearch output, or
  • make sure you have an index template that gets applied to the indexes you use and that maps fields according to your needs.
8

Can you give me an idea of index template I need to apply?

9

Will this work?

PUT _template/apache-template
{
"index_patterns" : ["apache*"],
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}

10

That looks like it should work.

Otherwise just use an index pattern of logstash-apache- in your pipeline.

11

Negative, this didn't work. "clientip" is the field containing requestor's ip address.

PUT _template/apache-template
{
"index_patterns" : ["apache*"],
"geoip" : {
"dynamic": true,
"properties" : {
"clientip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}

12

What's the name of the index you've been using for testing this? What do the actual mappings of that index look like? What does an example document look like?

13

Mapping...I have indexes for apache-access_YYYY.MM.DD and apache-error_YYYY.MM.DD. I tried to paste my mapping but got an error saying posts are limited to xxx characters. Preferred way to show you mapping?

14

Doc.

{

"_index": "apache-access_2017.12.10",
"_type": "doc",
"id": "zkvKPWABk5VwT0Rv22O",
"_version": 1,
"_score": null,
"_source": {
"request": "/",
"server": "aws9",
"agent": ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5"",
"source": "/var/log/httpd/access_log",
"logline": "8.8.8.8- - [10/Dec/2017:00:19:50 +0000] domiain.com "GET / HTTP/1.1" 200 17962 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5" Server=zws9 "TLSv1.2" 413255 0",
"request_duration_ms": 413255,
"clientip": "8.8.8.8",
"@version": "1",
"beat": {
"name": "aws9.domain.net",
"hostname": "aws9.domain.net",
"version": "6.0.0"
},
"host": "aws9.domian.net",
"virtualhost": "domain.com",
"ssl_protocol": "TLSv1.2",
"user_agent": {
"patch": "1",
"os": "Mac OS X",
"major": "11",
"minor": "0",
"build": "",
"os_minor": "13",
"os_major": "10",
"name": "Safari",
"os_name": "Mac OS X",
"device": "Other"
},
"timestamp": "10/Dec/2017:00:19:50 +0000",
"geoip": {
"timezone": "America/New_York",
"ip": "8.8.8.8",
"latitude": 39.9193,
"coordinates": [
-75.419,
39.9193
],
"continent_code": "NA",
"city_name": "Media",
"country_name": "United States",
"country_code2": "US",
"dma_code": 504,
"country_code3": "US",
"region_name": "Pennsylvania",
"location": {
"lon": -75.419,
"lat": 39.9193
},
"postal_code": "19063",
"region_code": "PA",
"longitude": -75.419
},
"offset": 24178299,
"verb": "GET",
"message": "8.8.8.8 - - [10/Dec/2017:00:19:50 +0000] domain.com "GET / HTTP/1.1" 200 17962 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5" Server=aws9 "TLSv1.2" 413255 0",
"tags": [
"beats_input_codec_plain_applied"
],
"referrer": ""-"",
"@timestamp": "2017-12-10T00:19:50.000Z",
"response": "200",
"bytes": 17962,
"httpversion": "1.1",
"request_duration_s": 0
},
"fields": {
"@timestamp": [
"2017-12-10T00:19:50.000Z"
]
},
"highlight": {
"clientip": [
"@kibana-highlighted-field@8.8.8.8@/kibana-highlighted-field@"
],
"virtualhost": [
"@kibana-highlighted-field@domain.com@/kibana-highlighted-field@"
]
},
"sort": [
1512865190000
]
}

15

I wouldn't mind doing that but I have tons of dashboards, visualizations and saved searches setup to use the current index pattern. If there was an easy way to change them I'd be all for it!

16

Given you are using a non-logstash based index pattern, you will need to edit the default logstash template and make sure the index pattern matches your custom one.

17

PUT _template/apache-template
{
"index_patterns" : ["apache*"],
"geoip" : {
"dynamic": true,
"properties" : {
"clientip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}

That's not a valid index template. See the example at Index templates | Elasticsearch Guide [8.11] | Elastic and compare the structure to what you have.

1 Like
18

Is this it?

PUT _template/apache-template
{
"index_patterns" : ["apache*"],
"mappings": {
"geoip" : {
"dynamic": true,
"properties" : {
"clientip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}

19

Not quite, unless "geoip" is the name of your document type. Study the example again, and note how the subitem of "mappings" is the document type.

20

I've looked over this a number of times on different occasions and don't see the issue, please help.