Logstash geoip

Elastic Stack Logstash
1

Hello,

I am trying to get the geoip working with my firewall

Logstash config, relevant part

udp {
    tags => "cisco-asa"
    type => "cisco-asa"
    port => 5047
    }

  geoip {

  source => "dst_ip"
  target => "geoip"
  #database => "/etc/logstash/GeoLiteCity.dat"
  #add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
  #add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
  }
  mutate {
  convert => [ "[geoip][coordinates]", "float"]
         }

else if [type] == "cisco-asa" {
    elasticsearch { hosts=> ["http://localhost:9200"]
    index => "cisco-asa-%{+YYYY.MM.dd}" #}
    stdout { codec => rubydebug }
   file { path => "/loggy/cisco-asa.log" }

When I check the cisco-asa index the geoip fields are created:
geoip.latitude number
geoip.location.lat number
geoip.location.lon number
geoip.longitude number
plus others (geoip.city_name, etc)

Default logstash containts these four only
geoip.ip ip
geoip.latitude number
geoip.location geo_point
geoip.longitude number

How do I add geoip.location with the geo_point type to cisco-asa index since it's missing?

This is what the file output looks like

{"src_interface":"inside","syslog_severity_code":5,"geoip":{"timezone":"America/Los_Angeles","ip":"XXXXXX","latitude":37.419200000000004,"continent_code":"NA","city_name":"Mountain View","country_name":"United States","country_code2":"US","dma_code":807,"country_code3":"US","region_name":"California","location":{"lon":-122.0574,"lat":37.419200000000004},"postal_code":"94043","region_code":"CA","longitude":-122.0574},"policy_id":"inside_access_in","syslog_facility":"user-level","syslog_facility_code":1,"message":"<164>Sep 21 2017 04:41:47: %ASA-4-106023: Deny tcp src inside:XXXXXXXX/62601 dst outside:XXXXXXX/5228 by access-group "inside_access_in" [0x0, 0x0]\n","type":"cisco-asa","dst_interface":"outside","syslog_severity":"notice","dst_ip":"XXXXXXXXX","tags":["cisco-asa","pre-processed","Firewall","ASA"],"src_ip":"XXXXXXXXX","src_port":"62601","protocol":"tcp","@timestamp":"2017-09-21T02:00:30.957Z","@version":"1","host":"XXXXXXXXXX","dst_port":"5228","action":"Deny","hashcode1":"0x0","hashcode2":"0x0"}

2

You have the two add_field sections commented out, so I would expect that you have geoip.location in your cisco-asa index, and not geoip.longitude.lat and geoip.longitude.lon. Can you please clarify exactly what is in the index?

Can you show us the mapping for the field?

3

I have tried with those two uncommented first as that was in the guide I was following, so it probably got created with .lat and .lon

How can I show you the mapping?

I was troubleshooting while you replied so I think I removed it with DELETE /_mapping/cisco-asa-date

I also commented out the geoip section, just confirmed there's no geoip fields mapped.

If I add the geoip section while leaving geoip.longitude.lat and geoip.longitude.lon still commented it should get populated with the right type?

Thank you!

EDIT:
I've added the geoip section to logstash config and I ended up with the same fields

Mapping looks like this, seems like it's missing the geo_point

Blockquote cisco-asa-2017.09.21": {
"mappings": {
"cisco-asa": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},

      "geoip": {
        "properties": {
          "city_name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "continent_code": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "country_code2": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "country_code3": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "country_name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "dma_code": {
            "type": "long"
          },
          "ip": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "latitude": {
            "type": "float"
          },
          "location": {
            "properties": {
              "lat": {
                "type": "float"
              },
              "lon": {
                "type": "float"
              }
            }
          },
          "longitude": {
            "type": "float"
          },
          "postal_code": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "region_code": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "region_name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "timezone": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
4

Have a read of https://www.elastic.co/blog/geoip-in-the-elastic-stack, it should provide some good guidance.

5

Thanks Mark
Once I changed the output to use logstash index it all started working.
I tried manually copying the logstash template and changing the name to cisco-asa and importing into es, however my indexes use cisco-asa-date as name so when thet get created they don't take the cisco-asa template settings, which was my original issue. It looks like logstash tries to guess the field types automatically since i dont have a template?
Is there a way to force that somehow.
I would just like to have them separate from the logstash indexes so i can easier track space usage.
What should I name the template so that anything with cisco-asa-* will use it
Would it be enough to just add the geo-map field types and the rest get created automatically?
Sorry for all these questions but I just struggle to understand it
I did go through the article you posted but will have another read

EDIT: So I tried importing the template again, basically I just copied the default logstash and change the template name. The issue first time is that the mapping also had to be change to my template name (cisco-asa)
Thanks again

Blockquote {
"template" : "cisco-asa-",
"version" : 50001,
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"cisco-asa" : {
"_all" : {"enabled" : true, "norms" : false},
"dynamic_templates" : [ {
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text",
"norms" : false
}
}
}, {
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "text", "norms" : false,
"fields" : {
"keyword" : { "type": "keyword", "ignore_above": 256 }
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date", "include_in_all": false },
"@version": { "type": "keyword", "include_in_all": false },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "half_float" },
"longitude" : { "type" : "half_float" }
}
}
}
}
}
}

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.