Cant get proper "type" : "ip"

Elastic Stack Elasticsearch
1

I've seen this around the internet, so I don't think I'm the only one having this issue. I send data from logstash -> elk

This is my config for logstash:
if [type] == "paloalto" and [SourceIP] !~ "(^127.0.0.1)|(^10.)|(^172.1[6-9].)|(^172.3[0-1].)|(^192.168..)|(^169.254.)" {

                    geoip {
                            source => "SourceIP"
                            target => "geoipsrc"
                            database => "/etc/logstash/GeoLiteCity.dat"
                            add_field => [ "[geoipsrc][coordinates]", "%{[geoipsrc][longitude]}" ]
                            add_field => [ "[geoipsrc][coordinates]", "%{[geoipsrc][latitude]}"  ]
                    }
                    mutate {
                            convert => [ "[geoipsrc][coordinates]", "float"]
                    }
            }

Elasticsearch gives me this error:

"_type"=>"paloalto", "_id"=>"AVN7-ovpvfjK8wUv_2e9", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [geoipsrc.ip]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: "93.189.26.18""}}}}, :level=>:warn}

This is my template;

it only errors out on "type": "ip" . Is this a bug?

"geoipsrc" : {
"properties" : {
"area_code" : {
"type" : "long"
},
"city_name" : {
"type" : "string",
"index" : "not_analyzed"
},
"continent_code" : {
"type" : "string",
"index" : "not_analyzed"
},
"coordinates" : {
"type" : "double"
},
"country_code2" : {
"type" : "string",
"index" : "not_analyzed"
},
"country_code3" : {
"type" : "string",
"index" : "not_analyzed"
},
"country_name" : {
"type" : "string",
"index" : "not_analyzed"
},
"dma_code" : {
"type" : "long"
},
"ip" : {
** "type" : "ip"**
** },**

2

If you take a message and then run it through with stdout + rubydebug, what does that field look like?

3

{
"message" => "Mar 9 22:18:48 1,2016/03/09 14:18:48,001801014079,TRAFFIC,end,1,2016/03/09 14:18:47,10.130.176.36,23.212.187.190,167.167.16.3,23.212.187.190,Permit_Trust_Out,,,ssl,vsys1,trust,untrust,ethernet1/3,ethernet1/2,Traffic Logs to GISO,2016/03/09 14:18:48,186042,1,40489,443,60197,443,0x404000,tcp,allow,2866,1985,881,15,2016/03/09 14:16:17,121,any,0,1865515826,0x0,10.0.0.0-10.255.255.255,United States,0,9,6",
"@version" => "1",
"@timestamp" => "2016-03-16T18:03:24.955Z",
"path" => "/etc/logstash/paloalto/paa.log",
"host" => "ukequlvpoc003",
"type" => "paloalto",
"Domain" => "Mar 9 22:18:48 1",
"ReceiveTime" => "2016/03/09 14:18:48",
"Serial" => "001801014079",
"Type" => "TRAFFIC",
"ThreatContentType" => "end",
"ConfigVersion" => "1",
"GenerateTime" => "2016/03/09 14:18:47",
"SourceIP" => "10.130.176.36",
"DestinationIP" => "23.212.187.190",
"NATSourceIP" => "167.167.16.3",
"NATDestinationIP" => "23.212.187.190",
"Rule" => "Permit_Trust_Out",
"SourceUser" => nil,
"DestinationUser" => nil,
"Application" => "ssl",
"VirtualSystem" => "vsys1",
"SourceZone" => "trust",
"DestinationZone" => "untrust",
"InboundInterface" => "ethernet1/3",
"OutboundInterface" => "ethernet1/2",
"LogAction" => "Traffic Logs to GISO",
"TimeLogged" => "2016/03/09 14:18:48",
"SessionID" => "186042",
"RepeatCount" => "1",
"SourcePort" => "40489",
"DestinationPort" => "443",
"NATSourcePort" => "60197",
"NATDestinationPort" => "443",
"Flags" => "0x404000",
"Protocol" => "tcp",
"Action" => "allow",
"Bytes" => "2866",
"BytesSent" => "1985",
"BytesReceived" => "881",
"Packets" => "15",
"StartTime" => "2016/03/09 14:16:17",
"ElapsedTime" => "121",
"Category" => "any",
"Padding" => "0",
"Seqnum" => "1865515826",
"Actionflags" => "0x0",
"SourceCountry" => "10.0.0.0-10.255.255.255",
"DestinationCountry" => "United States",
"Cpadding" => "0",
"PktsSent" => "9",
"PktsReceived" => "6",
"tags" => [
[0] "pa-traffic"
],
"geoipdst" => {
"ip" => "23.212.187.190",
"country_code2" => "US",
"country_code3" => "USA",
"country_name" => "United States",
"continent_code" => "NA",
"region_name" => "MA",
"city_name" => "Cambridge",
"postal_code" => "02142",
"latitude" => 42.362599999999986,
"longitude" => -71.0843,
"dma_code" => 506,
"area_code" => 617,
"timezone" => "America/New_York",
"real_region_name" => "Massachusetts",
"location" => [
[0] -71.0843,
[1] 42.362599999999986
],
"coordinates" => [
[0] -71.0843,
[1] 42.362599999999986
]
}
}

4

Here's another one broken down really simple;

{
"paloalto-2016.03.16": {
"aliases": { },
"mappings": {
"paloalto": { },
"tweet": {
"properties": {
"Domain": {
"type": "ip"
},
"message": {
"type": "string"
}
}
}
},
"settings": {
"index": {
"creation_date": "1458153301420",
"uuid": "oJaZ0OHTS86lhIsgm3BlNg",
"number_of_replicas": "1",
"number_of_shards": "1",
"version": {
"created": "2020199"
}
}
},
"warmers": { }
}
}

5

{
"message" => "192.168.1.6",
"@version" => "1",
"@timestamp" => "2016-03-16T18:39:20.148Z",
"path" => "/etc/logstash/paloalto/patest.log",
"host" => "ukequlvpoc003",
"type" => "paloalto",
"Domain" => "192.168.1.6",
"tags" => [
[0] "pa-threats"
]
}

6

The answer is in this thread GeoIP location has added brackets and not able to visualize.

Basically, you are using the wrong config for geoip. There is no need to use mutate plugin with geoip plugin. When you use geoip plugin, it will output a bunch of fields, and location, one of the fields, has geo_point format and contains the info needed for Kibana to graph Tile maps. You just need a mapping template for the geoip fields in ES.

Below is what the mapping looks like:

"geoip": {
          "dynamic": true,
          "type": "object",
          "properties": {
            "city_name": {
              "index": "not_analyzed",
              "type": "string"
            },
            "timezone": {
              "index": "not_analyzed",
              "type": "string"
            },
            "country_code2": {
              "index": "not_analyzed",
              "type": "string"
            },
            "country_name": {
              "index": "not_analyzed",
              "type": "string"
            },
            "continent_code": {
              "index": "not_analyzed",
              "type": "string"
            },
            "location": {
              "type": "geo_point",
              "doc_values": true
            },
            "region_name": {
              "index": "not_analyzed",
              "type": "string"
            },
            "real_region_name": {
              "index": "not_analyzed",
              "type": "string"
            },
            "postal_code": {
              "index": "not_analyzed",
              "type": "string"
            }
          }
        },

geoip.location is the field for Tile map in Kibana.

7

Nope.

If input a file with one ip address "192.168.1.1"

and then I create a template with "domain":"ip"

it says that this is not the right type

Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"paloalto-2016.03.16", :_type=>"paloalto", :_routing=>nil}, #<LogStash::Event:0x530f722e @metadata={"path"=>"/etc/logstash/paloalto/patest.log"}, @accessors=#<LogStash::Util::Accessors:0xec9dba6 @store={"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, @lut={"path"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "path"], "host"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "host"], "type"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "type"], "[type]"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "type"], "[message]"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "message"], "message"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "message"], "[Domain]"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "Domain"], "tags"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "tags"], "[SourceIP]"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "SourceIP"], "SourceIP"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "SourceIP"]}>, @data={"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, @metadata_accessors=#<LogStash::Util::Accessors:0x7fa5cd21 @store={"path"=>"/etc/logstash/paloalto/patest.log"}, @lut={"[path]"=>[{"path"=>"/etc/logstash/paloalto/patest.log"}, "path"]}>, @cancelled=false>], :response=>{"create"=>{"_index"=>"paloalto-2016.03.16", "_type"=>"paloalto", "_id"=>"AVOAtaZj6FFiT7UMT_ZE", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [Domain]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: "192.168.1.5""}}}}, :level=>:warn}

this isnt normal behavior...