Logstash Geoip filter with Packetbeat

[jkevan (Josh)]

Hi everyone,
to start, I'm running Logstash, elasticsearch and kibana 6.3.1 and multiple beats that are all on 6.3.2. I have loaded the various default beats index templates as outlined in https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-template.html (as well as auditbeat, metricbeat, and winlogbeat). all of my beats are using the Logstash output.

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.x.x.x:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

The issue I am having is that i cannot get the Geoip filter to work with packetbeat. Here is my .conf file that handles the beats input

# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
    beats {
        type => beats
        port => 5044
        ssl => false
    }
}

filter {
    geoip {
       source => "[dest][ip]"
       target => "[dest][ip_location]"
       }


    geoip {
        source => "[source][ip]"
        target => "[source][ip_location]"
        }
}

output {
    elasticsearch {
        hosts => ["10.x.x.x:9200", "10.x.x.x:9200", "10.x.x.x:9200"]
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        manage_template => false
        }
}

I'm pretty sure I'm using the geoip filter wrong, but I'm not sure how. If you need more info please let me know, and Thank you for all that you do!

[Badger]

Do you get any errors or do you just not get a geo_point in Kibana? If it is the latter then you need an index template. See this thread.

[jkevan (Josh)]

I get no error, but i setup my template from the installation
directions:https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-template.html

here is the current index setup: https://gist.github.com/jkevan91/2a96da1a46757b489691b334109d8d88

[jkevan (Josh)]

I take it back i do get an error in logstash:

 {"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"parse_exception", "reason"=>"field [lat] missing"}}}}}
[2018-08-10T11:16:10,830][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"packetbeat-6.3.2-2018.08.10", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x681ebb0e>], :response=>{"index"=>{"_index"=>"packetbeat-6.3.2-2018.08.10", "_type"=>"doc", "_id"=>"vl4NJWUBa96CdXgWkW8T", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"parse_exception", "reason"=>"field [lat] missing"}}}}}

[Badger]

@jkevan

"reason"=>"field [lat] missing"

OK. So your template defines several fields as geo_point. It would appear your event has one of those fields, but it does not have a lat/lon pair of numbers. The error message in elasticsearch may show you more information, or else dump it to

output { stdout { codec=> rubydebug } }