I'm trying to use Filebeat input to create tile maps of http access. The geoip data are captured, but when I try and create a map, I get the following error:
No Compatible Fields: The "filebeat-*" index pattern does not contain any of the following field types: geo_point
Can I correct this? How?
Thanks.
How does your log file look like? Filebeat does not do any log processing. If you want to split up your log lines into fields you need to use Logstash.
ruflin,
Sorry, I don't understand your response. I do use Logstash to split up my log lines.
Here's the filter I'm using to get geoip data:
filter {
if [type] == "apache-access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
locale => "en"
target => "@timestamp"
}
geoip {
source => "clientip"
database => "/etc/logstash/GeoLiteCity.dat"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
add_tag => "geoip"
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
In order to keep my beats indices separate (especially Topbeat and Packetbeat), I've set up output like this:
output {
if [@metadata][beat] {
elasticsearch {
hosts => ["http://10.0.101.101:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else {
elasticsearch { hosts => ["10.0.101.101:9200"] }
stdout { codec => rubydebug }
}
}
That gives me the following indices: logstash-, filebeat-, topbeat-, packetbeat-, ... . I can see geoip date in various filebeat log captures, as in the following (sorry for the length of this):
{
"_index": "filebeat-2016.08.26",
"_type": "apache-access",
"_id": "AVbHcdzSCSz4NTRp6VCx",
"_score": null,
"_source": {
"message": "158.130.6.191 - - [26/Aug/2016:11:22:01 -0400] "HEAD /images/ HTTP/1.1" 404 - "-" "Mozilla/5.0 zgrab/0.x"",
"@version": "1",
"@timestamp": "2016-08-26T15:22:01.000Z",
"input_type": "log",
"fields": null,
"type": "apache-access",
"beat": {
"hostname": "myhost.com",
"name": "myhost.com"
},
"source": "/var/log/httpd/access_log",
"offset": 5621849,
"count": 1,
"host": "myhost",
"tags": [
"log",
"beats_input_codec_plain_applied",
"geoip"
],
"clientip": "x.x.x.x",
"ident": "-",
"auth": "-",
"timestamp": "26/Aug/2016:11:22:01 -0400",
"verb": "HEAD",
"request": "/images/",
"httpversion": "1.1",
"response": "404",
"referrer": ""-"",
"agent": ""Mozilla/5.0 zgrab/0.x"",
"geoip": {
"ip": "y.y.y.y",
"country_code2": "US",
"country_code3": "USA",
"country_name": "United States",
"continent_code": "NA",
"region_name": "PA",
"city_name": "Philadelphia",
"postal_code": "19104",
"latitude": 39.9597,
"longitude": -75.1968,
"dma_code": 504,
"area_code": 215,
"timezone": "America/New_York",
"real_region_name": "Pennsylvania",
"location": [
-75.1968,
39.9597
],
"coordinates": [
-75.1968,
39.9597
]
}
},
"fields": {
"@timestamp": [
1472224921000
]
},
"highlight": {
"tags": [
"@kibana-highlighted-field@geoip@/kibana-highlighted-field@"
]
},
"sort": [
1472224921000
]
}
Great. Now how do I visualize this in Kibana? As stated in the OP, I get "No Compatible Fields: The "filebeat-" index pattern does not contain any of the following field types: geo_point" if I use the filebeat- indices, and no output if I use logstash-* indices.
I suggest you edit your post and change its category to Logstash since this is a Logstash question.
Logstash's default index template which would've taken care of mapping your GeoIP field as geo_point only applies to logstash-* indexes. You need to install another index template (or modify the existing template) to also apply to filebeat-* indexes.
Thanks, Magnus.
I used the command "curl -XGET 'http://localhost:9200/_template/logstash?pretty' > /tmp/logstash.template", and found this in logstash.template:
{
"logstash" : {
"order" : 0,
"template" : "logstash-*",
"settings" : {
"index" : {
"refresh_interval" : "5s"
}
},
If I wanted to add filebeat-*, would it be:
"template" : "logstash-*" "filebeat-*",
and, if so, what does that do to my current logstash and filebeat output, if anything?
I didn't find a filebeat template, at least not with the curl command above, substituting in "filebeat?". Should there be a filebeat template? If not, how would I create one (copy logstash template, with appropriate changes?), and would it change or break the filebeat output I'm seeing now? Btw, I find in the logstash template:
}, {
"geo_point_fields" : {
"mapping" : {
"type" : "geo_point",
"doc_values" : true
},
"match_mapping_type" : "geo_point",
"match" : "*"
}
} ],
"_all" : {
"omit_norms" : true,
"enabled" : true
},
"properties" : {
"@timestamp" : {
"type" : "date",
"doc_values" : true
},
"geoip" : {
"dynamic" : true,
"type" : "object",
"properties" : {
"ip" : {
"type" : "ip",
"doc_values" : true
},
"latitude" : {
"type" : "float",
"doc_values" : true
},
"location" : {
"type" : "geo_point",
"doc_values" : true
},
"longitude" : {
"type" : "float",
"doc_values" : true
}
}
},
I guess the geo_point part is what I'm after.
Thank you.
If I wanted to add filebeat-*, would it be:
"template" : "logstash-*" "filebeat-*",
No, because that's not valid JSON. I don't think a single template can include multiple index name patterns so you'll have to install a separate template.
I didn't find a filebeat template, at least not with the curl command above, substituting in "filebeat?". Should there be a filebeat template?
Logstash isn't creating it for you, so no. You'll have to create it yourself. However, Logstash's elasticsearch output can manage it for you.
and would it change or break the filebeat output I'm seeing now?
It shouldn't break anything, but fields will be mapped differently so you'll see different behavior in Kibana.
I guess the geo_point part is what I'm after.
Yes.
I'm running into the same issue. I'm using a filebeat input and am adding in the geoip location information successfully. When trying to generate the tile map, I receive the message No Compatible Fields: The "filebeat-*" index pattern does not contain any of the following field types: geo_point but am unsure how to get this going.
Looking at the logstash-* index geoip.location shows as geopoint but filebeat-* index shows the same as just number However running: curl -XGET 'http://localhost:9200/_template/filebeat?pretty' shows:
{
"filebeat" : {
"order" : 0,
"template" : "filebeat-",
"settings" : {
"index" : {
"refresh_interval" : "5s"
}
},
"mappings" : {
"default" : {
"dynamic_templates" : [ {
"template1" : {
"mapping" : {
"ignore_above" : 1024,
"index" : "not_analyzed",
"type" : "{dynamic_type}",
"doc_values" : true
},
"match" : ""
}
} ],
"_all" : {
"norms" : {
"enabled" : false
},
"enabled" : true
},
"properties" : {
"@timestamp" : {
"type" : "date"
},
"geoip" : {
"dynamic" : true,
"type" : "object",
"properties" : {
"location" : {
"type" : "geo_point"
}
}
},
"offset" : {
"type" : "long",
"doc_values" : "true"
},
"message" : {
"index" : "analyzed",
"type" : "string"
}
}
}
},
"aliases" : { }
}
}
I've refreshed the index and well as deleted and re-created but still don't have the ability to use a geohash. I'm sure it is something simple!
If I import filebeat.template.json from a server running Filebeat to my ELK server, edit that file to include the geo_point part, and load the template with "curl -XPOST 'http://localhost:9200/_template/filebeat?pretty' -d @/home/me/filebeat.template.json", will that get the tile maps working with filebeat? If so, what, exactly, do I put in filebeat.template.json to make it so.
Thanks.
The template you posted previously should work with the event you posted even earlier. If it doesn't work, please post the mappings of an actual index (so that we can see that the template has been applied) and an example document from ES that contains the geoip field.
Magnus,
Sorry, I'm thick as a brick, and hope I'm not being too exasperating.
When you say "The template you posted previously ... ", are you referring to the template that rsaeks actually posted? The only template that is loaded in my ELK is the Logstash template. Don't I need a Filebeat template?
Diggy
Oh, sorry. Yes, you need an index template that matches Filebeat indexes and that maps your [geoip][coordinates] field as geo_point. You can use the Logstash template as a starting point.
OK. However, since I don't have a real good idea as to how to create the proper template, would this work?:
{
"mappings": {
"_default_": {
"_all": {
"enabled": true,
"norms": {
"enabled": false
}
}, {
"geo_point_fields" : {
"mapping" : {
"type" : "geo_point",
"doc_values" : true
},
"match_mapping_type" : "geo_point",
"match" : "*"
}
} ],
"_all" : {
"omit_norms" : true,
"enabled" : true
},
"dynamic_templates": [
{
"template1": {
"mapping": {
"doc_values": true,
"ignore_above": 1024,
"index": "not_analyzed",
"type": "{dynamic_type}"
},
"match": "*"
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"message": {
"type": "string",
"index": "analyzed"
},
"offset": {
"type": "long",
"doc_values": "true"
}
}
}
},
"settings": {
"index.refresh_interval": "5s"
},
"template": "filebeat-*"
}Almost. You have a block that says what settings to apply to fields that indeed are mapped as geo_point, but there's nothing that actually maps a field as geo_point. You need to add something like
"geoip": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
to your current "properties" block if you want [geoip][coordinates] to be a geo_point field.
I created this template, and loaded it with "curl -XPOST 'http://local:9200/_template/filebeat?pretty' -d @/home/me/filebeat.template.json" on my ELK server:
{
"mappings": {
"_default_": {
"_all": {
"enabled": true,
"norms": {
"enabled": false
}
},
"dynamic_templates": [
{
"template1": {
"mapping": {
"doc_values": true,
"ignore_above": 1024,
"index": "not_analyzed",
"type": "{dynamic_type}"
},
"match": "*"
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"message": {
"type": "string",
"index": "analyzed"
},
"offset": {
"type": "long",
"doc_values": "true"
},
"geoip": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}
},
"settings": {
"index.refresh_interval": "5s"
},
"template": "filebeat-*"
}
I got:
{
"acknowledged" : true
}
in return. However, I still get: "No Compatible Fields: The "filebeat-*" index pattern does not contain any of the following field types: geo_point" in Kibana. Arrrgh! Do I still have a misconfiguration?
Diggy
Since creating the template, did you create a new index and populate it with data? Have you verified that the created index gets the expected mappings? Did you refresh the field list in Kibana?
This topic was automatically closed after 21 days. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.