Cant get proper "type" : "ip"


(Blake Wolfson) #1

I've seen this around the internet, so I don't think I'm the only one having this issue. I send data from logstash -> elk

This is my config for logstash:
if [type] == "paloalto" and [SourceIP] !~ "(^127.0.0.1)|(^10.)|(^172.1[6-9].)|(^172.3[0-1].)|(^192.168..)|(^169.254.)" {

                    geoip {
                            source => "SourceIP"
                            target => "geoipsrc"
                            database => "/etc/logstash/GeoLiteCity.dat"
                            add_field => [ "[geoipsrc][coordinates]", "%{[geoipsrc][longitude]}" ]
                            add_field => [ "[geoipsrc][coordinates]", "%{[geoipsrc][latitude]}"  ]
                    }
                    mutate {
                            convert => [ "[geoipsrc][coordinates]", "float"]
                    }
            }

Elasticsearch gives me this error:

"_type"=>"paloalto", "_id"=>"AVN7-ovpvfjK8wUv_2e9", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [geoipsrc.ip]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: "93.189.26.18""}}}}, :level=>:warn}

This is my template;

it only errors out on "type": "ip" . Is this a bug?

"geoipsrc" : {
"properties" : {
"area_code" : {
"type" : "long"
},
"city_name" : {
"type" : "string",
"index" : "not_analyzed"
},
"continent_code" : {
"type" : "string",
"index" : "not_analyzed"
},
"coordinates" : {
"type" : "double"
},
"country_code2" : {
"type" : "string",
"index" : "not_analyzed"
},
"country_code3" : {
"type" : "string",
"index" : "not_analyzed"
},
"country_name" : {
"type" : "string",
"index" : "not_analyzed"
},
"dma_code" : {
"type" : "long"
},
"ip" : {
** "type" : "ip"**
** },**


(Mark Walkom) #2

If you take a message and then run it through with stdout + rubydebug, what does that field look like?


(Blake Wolfson) #3

{
"message" => "Mar 9 22:18:48 1,2016/03/09 14:18:48,001801014079,TRAFFIC,end,1,2016/03/09 14:18:47,10.130.176.36,23.212.187.190,167.167.16.3,23.212.187.190,Permit_Trust_Out,,,ssl,vsys1,trust,untrust,ethernet1/3,ethernet1/2,Traffic Logs to GISO,2016/03/09 14:18:48,186042,1,40489,443,60197,443,0x404000,tcp,allow,2866,1985,881,15,2016/03/09 14:16:17,121,any,0,1865515826,0x0,10.0.0.0-10.255.255.255,United States,0,9,6",
"@version" => "1",
"@timestamp" => "2016-03-16T18:03:24.955Z",
"path" => "/etc/logstash/paloalto/paa.log",
"host" => "ukequlvpoc003",
"type" => "paloalto",
"Domain" => "Mar 9 22:18:48 1",
"ReceiveTime" => "2016/03/09 14:18:48",
"Serial" => "001801014079",
"Type" => "TRAFFIC",
"ThreatContentType" => "end",
"ConfigVersion" => "1",
"GenerateTime" => "2016/03/09 14:18:47",
"SourceIP" => "10.130.176.36",
"DestinationIP" => "23.212.187.190",
"NATSourceIP" => "167.167.16.3",
"NATDestinationIP" => "23.212.187.190",
"Rule" => "Permit_Trust_Out",
"SourceUser" => nil,
"DestinationUser" => nil,
"Application" => "ssl",
"VirtualSystem" => "vsys1",
"SourceZone" => "trust",
"DestinationZone" => "untrust",
"InboundInterface" => "ethernet1/3",
"OutboundInterface" => "ethernet1/2",
"LogAction" => "Traffic Logs to GISO",
"TimeLogged" => "2016/03/09 14:18:48",
"SessionID" => "186042",
"RepeatCount" => "1",
"SourcePort" => "40489",
"DestinationPort" => "443",
"NATSourcePort" => "60197",
"NATDestinationPort" => "443",
"Flags" => "0x404000",
"Protocol" => "tcp",
"Action" => "allow",
"Bytes" => "2866",
"BytesSent" => "1985",
"BytesReceived" => "881",
"Packets" => "15",
"StartTime" => "2016/03/09 14:16:17",
"ElapsedTime" => "121",
"Category" => "any",
"Padding" => "0",
"Seqnum" => "1865515826",
"Actionflags" => "0x0",
"SourceCountry" => "10.0.0.0-10.255.255.255",
"DestinationCountry" => "United States",
"Cpadding" => "0",
"PktsSent" => "9",
"PktsReceived" => "6",
"tags" => [
[0] "pa-traffic"
],
"geoipdst" => {
"ip" => "23.212.187.190",
"country_code2" => "US",
"country_code3" => "USA",
"country_name" => "United States",
"continent_code" => "NA",
"region_name" => "MA",
"city_name" => "Cambridge",
"postal_code" => "02142",
"latitude" => 42.362599999999986,
"longitude" => -71.0843,
"dma_code" => 506,
"area_code" => 617,
"timezone" => "America/New_York",
"real_region_name" => "Massachusetts",
"location" => [
[0] -71.0843,
[1] 42.362599999999986
],
"coordinates" => [
[0] -71.0843,
[1] 42.362599999999986
]
}
}


(Blake Wolfson) #4

Here's another one broken down really simple;

{
"paloalto-2016.03.16": {
"aliases": { },
"mappings": {
"paloalto": { },
"tweet": {
"properties": {
"Domain": {
"type": "ip"
},
"message": {
"type": "string"
}
}
}
},
"settings": {
"index": {
"creation_date": "1458153301420",
"uuid": "oJaZ0OHTS86lhIsgm3BlNg",
"number_of_replicas": "1",
"number_of_shards": "1",
"version": {
"created": "2020199"
}
}
},
"warmers": { }
}
}


(Blake Wolfson) #5

{
"message" => "192.168.1.6",
"@version" => "1",
"@timestamp" => "2016-03-16T18:39:20.148Z",
"path" => "/etc/logstash/paloalto/patest.log",
"host" => "ukequlvpoc003",
"type" => "paloalto",
"Domain" => "192.168.1.6",
"tags" => [
[0] "pa-threats"
]
}


(Anh) #6

The answer is in this thread GeoIP location has added brackets and not able to visualize.

Basically, you are using the wrong config for geoip. There is no need to use mutate plugin with geoip plugin. When you use geoip plugin, it will output a bunch of fields, and location, one of the fields, has geo_point format and contains the info needed for Kibana to graph Tile maps. You just need a mapping template for the geoip fields in ES.

Below is what the mapping looks like:

"geoip": {
          "dynamic": true,
          "type": "object",
          "properties": {
            "city_name": {
              "index": "not_analyzed",
              "type": "string"
            },
            "timezone": {
              "index": "not_analyzed",
              "type": "string"
            },
            "country_code2": {
              "index": "not_analyzed",
              "type": "string"
            },
            "country_name": {
              "index": "not_analyzed",
              "type": "string"
            },
            "continent_code": {
              "index": "not_analyzed",
              "type": "string"
            },
            "location": {
              "type": "geo_point",
              "doc_values": true
            },
            "region_name": {
              "index": "not_analyzed",
              "type": "string"
            },
            "real_region_name": {
              "index": "not_analyzed",
              "type": "string"
            },
            "postal_code": {
              "index": "not_analyzed",
              "type": "string"
            }
          }
        },

geoip.location is the field for Tile map in Kibana.


(Blake Wolfson) #7

Nope.

If input a file with one ip address "192.168.1.1"

and then I create a template with "domain":"ip"

it says that this is not the right type

Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"paloalto-2016.03.16", :_type=>"paloalto", :_routing=>nil}, #<LogStash::Event:0x530f722e @metadata={"path"=>"/etc/logstash/paloalto/patest.log"}, @accessors=#<LogStash::Util::Accessors:0xec9dba6 @store={"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, @lut={"path"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "path"], "host"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "host"], "type"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "type"], "[type]"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "type"], "[message]"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "message"], "message"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "message"], "[Domain]"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "Domain"], "tags"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "tags"], "[SourceIP]"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "SourceIP"], "SourceIP"=>[{"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, "SourceIP"]}>, @data={"message"=>"192.168.1.5", "@version"=>"1", "@timestamp"=>"2016-03-16T18:35:00.610Z", "path"=>"/etc/logstash/paloalto/patest.log", "host"=>"ukequlvpoc003", "type"=>"paloalto", "Domain"=>"192.168.1.5", "tags"=>["pa-threats"]}, @metadata_accessors=#<LogStash::Util::Accessors:0x7fa5cd21 @store={"path"=>"/etc/logstash/paloalto/patest.log"}, @lut={"[path]"=>[{"path"=>"/etc/logstash/paloalto/patest.log"}, "path"]}>, @cancelled=false>], :response=>{"create"=>{"_index"=>"paloalto-2016.03.16", "_type"=>"paloalto", "_id"=>"AVOAtaZj6FFiT7UMT_ZE", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [Domain]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: "192.168.1.5""}}}}, :level=>:warn}

this isnt normal behavior...


(system) #8