Getting "reason"=>"geo_point expected" error from geoip whilst indexing


(Thomas Knowles) #1

I have filebeat running on one of my servers which collects iptables and telnet logs, both feeds use the geoip filter plugin and both feed into the logstash for parsing and then subsequently into elasticsearch for storing. However, I am getting the following error when trying to index the telnet logs though the iptables ones work fine:

All running 5.0.1

Error on Logstash:

[2016-11-22T11:07:08,956][WARN ][logstash.outputs.elasticsearch] Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"telnet-2016.11.22", :_type=>"telnet", :_routing=>nil}, 2016-11-22T11:06:59.000Z lon-01-collect-001 %{message}], :response=>{"index"=>{"_index"=>"telnet-2016.11.22", "_type"=>"telnet", "_id"=>"AViLt88Y87jAoXX9m1im", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"parse_exception", "reason"=>"geo_point expected"}}}}}

filebeat.yml

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/iptables.log
  document_type: iptables
- input_type: log
  paths:
    - /var/log/telnet.log
  document_type: telnet
output.logstash:
  hosts: ["MY_LOGSTASH_SERVER:5044"]

logstashconfig.conf

input {
  beats {
    port => 5044
  }
}
filter {
  if [type] == "iptables" {
    grok {
      patterns_dir => ["/etc/logstash/patterns/"]
      match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{SYSLOGHOST} %{SYSLOGPROG} IPTables-Dropped: %{IPTABLES}" }
    }
    date {
      match => [ "log_timestamp", "ISO8601" ]
    }
  } else if [type] == "telnet" {
    grok {
      match => { "message" => "%{INT:log_timestamp},%{IP:src_ip},%{WORD:username},%{WORD:password}" }
    }
    date {
      match => [ "log_timestamp", "UNIX" ]
    }
  }
  geoip {
    source => "src_ip"
  }
  mutate {
    remove_field => [ "log_timestamp", "message" ]
  }
}
output {
  if [type] == "iptables" {
    elasticsearch {
      hosts => [ "MY_ELASTICSEARH_SERVER:9200" ]
      index => [ "iptables-%{+YYYY.MM.dd}" ]
      manage_template => false
    }
  } else if [type] == "telnet" {
    elasticsearch {
      hosts => [ "MY_ELASTICSEARH_SERVER:9200" ]
      index => [ "telnet-%{+YYYY.MM.dd}" ]
      manage_template => false
    }
  }
}

Sample telnet log:

1479813223,220.133.42.168,admin,xmhdipc

Logstash file output of the log:

{
	"geoip": {
		"timezone": null,
		"ip": "81.19.139.197",
		"latitude": 55.7386,
		"continent_code": "EU",
		"city_name": null,
		"country_code2": "RU",
		"country_name": "Russia",
		"dma_code": null,
		"country_code3": "RU",
		"region_name": null,
		"location": [37.6068, 55.7386],
		"postal_code": null,
		"longitude": 37.6068,
		"region_code": null
	},
	"offset": 346958,
	"input_type": "log",
	"source": "/var/log/telnet.log",
	"type": "telnet",
	"tags": ["beats_input_codec_plain_applied"],
	"src_ip": "81.19.139.197",
	"password": "1111",
	"@timestamp": "2016-11-22T11:18:17.000Z",
	"@version": "1",
	"beat": {
		"hostname": "lon-01-collect-001",
		"name": "lon-01-collect-001",
		"version": "5.0.1"
	},
	"host": "lon-01-collect-001",
	"username": "root"
}

(Thomas Knowles) #2

Exceeded body count (continued):

mapping in elasticsearch:

curl -XGET 'localhost:9200/telnet-2016.11.22/_mapping?pretty'
{
  "telnet-2016.11.22" : {
    "mappings" : {
      "iptables" : {
        "dynamic" : "false",
        "_all" : {
          "enabled" : false
        },
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "geoip" : {
            "properties" : {
              "city_name" : {
                "type" : "keyword"
              },
              "continent_code" : {
                "type" : "keyword"
              },
              "country_code2" : {
                "type" : "keyword"
              },
              "country_code3" : {
                "type" : "keyword"
              },
              "country_name" : {
                "type" : "keyword"
              },
              "dma_code" : {
                "type" : "long"
              },
              "ip" : {
                "type" : "ip"
              },
              "latitude" : {
                "type" : "double"
              },
              "location" : {
                "type" : "geo_point"
              },
              "longitude" : {
                "type" : "double"
              },
              "postal_code" : {
                "type" : "keyword"
              },
              "region_code" : {
                "type" : "keyword"
              },
              "region_name" : {
                "type" : "keyword"
              },
              "timezone" : {
                "type" : "keyword"
              }
            }
          },
          "password" : {
            "type" : "keyword"
          },
          "src_ip" : {
            "type" : "ip"
          },
          "username" : {
            "type" : "keyword"
          }
        }
      }
    }
  }
}

Elasticsearch error:

[2016-11-22T13:47:54,536][DEBUG][o.e.a.b.TransportShardBulkAction] [fra-01-store-001] [telnet-2016.11.22][1] failed to execute bulk item (index) index {[telnet-2016.11.22][telnet][AViMSv0h87jAoXX9m2NV], source[{"geoip":{"timezone":"America/New_York","ip":"207.68.114.21","latitude":40.5536,"continent_code":"NA","city_name":"Altoona","country_code2":"US","country_name":"United States","dma_code":574,"country_code3":"US","region_name":"Pennsylvania","location":[-78.424,40.5536],"postal_code":"16601","longitude":-78.424,"region_code":"PA"},"offset":387310,"input_type":"log","source":"/var/log/telnet.log","type":"telnet","tags":["beats_input_codec_plain_applied"],"src_ip":"207.68.114.21","password":"xmhdipc","@timestamp":"2016-11-22T13:47:52.000Z","@version":"1","beat":{"hostname":"lon-01-collect-001","name":"lon-01-collect-001","version":"5.0.1"},"host":"lon-01-collect-001","username":"root"}]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
	at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:175) ~[elasticsearch-5.0.1.jar:5.0.1]
.... Lot's of "ats"
Caused by: org.elasticsearch.ElasticsearchParseException: geo_point expected
   .... More "ats"
... 33 more

Any ideas?


(Thomas Knowles) #3

In my mapping I had referenced the wrong type:

"mappings":{
"telnet":{
"dynamic": false,

Item highlighted was the wrong value it was previously iptables.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.