Unable to put dstip_geoip.location on map


(Frank Trezza) #1

In my Logstash.conf file I have the following to parse my sonicwall logs, which is working well and I am getting good data thats geo-coded on the fly, but when using kibana and trying to create a tile map the field doesn't show up. How do I go about getting this field in there? My logstash config is below.

#logstash.conf

input {
udp {
type => sonicwall
codec => plain {
charset => "ISO-8859-1"
}
port => 514
}
}

filter {
if [type] == "sonicwall" {

kv {
    exclude_keys => [ "<129>id", "<132>id", "<133>id","<134>id", "af_polid", "af_service", "app", "appid", "code", "fw", "m", "op", "sn" ]
}

date {
    match => [ "time", "yyyy-MM-dd HH:mm:ss z", "yyyy-MM-dd HH:mm:ss" ]
}

if [src] {
    grok {
        match => { 
            "src" => [
                "%{IP:srcip}:%{INT:srcport}:%{DATA:srcint}:%{GREEDYDATA:srcname}",
                "%{IP:srcip}:%{INT:srcport}:%{DATA:srcint}",
                "%{IP:srcip}::%{DATA:srcint}",
                ":%{INT:srcport}"
            ]  
        }
    }
}

if [dst] {
    grok {
        match => { 
            "dst" => [ 
                "%{IP:dstip}:%{INT:dstport}:%{DATA:dstint}:%{GREEDYDATA:dstinfo}",
                "%{IP:dstip}:%{INT:dstport}:%{DATA:dstint}",
                "%{IP:dstip}::%{DATA:dstint}",
                ":%{INT:dstport}"
            ]
        }
    }
}

# Sanitize fields with \r after recent firmware update
mutate {
    gsub => [
        "sent", "\r", "",
        "rcvd", "\r", "",
        "cdur", "\r", "",
        "spkt", "\r", "",
        "rpkt", "\r", "",
        "proto", "\r", ""
    ]
}

# Assign network tags based on IP
if [dstip] {
    cidr {
        add_tag => ["dstip-private"]
        address => ["%{dstip}"]
        network => ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]
    }                
    
    # Parse GeoIP info
    if "dstip-private" not in [tags] {
        geoip {
            source => "dstip"
            target => "dstip_geoip"
            fields => ["country_name", "region_name", "city_name", "location"]
               }
    }
}

# Replace srcname with srcip if srcname does not exist
if ![srcname] and [srcip] {
    mutate {
        replace => { "srcname" => "%{srcip}" }
    }
}	

# Replace dstname with dstinfo or dstip if dstname does not exist
if ![dstname] and [dstinfo] {
    mutate {
        replace => { "dstname" => "%{dstinfo}" }
    }
} else if ![dstname] and [dstip] {
    mutate {
        replace => { "dstname" => "%{dstip}" }
    }
}	

mutate {
    lowercase => [ "msg", "appName", "sess", "fw_action", "srcint", "dstint", "Category"]
    remove_field => [ "src", "dst", "dstinfo", "message", "time" ]
    
}	    

}
}

output {
elasticsearch { hosts => ["localhost:9201"] }
stdout { codec => rubydebug }
}


(Lee Drengenberg) #2

Hi Frank,

If you look at your index pattern fields in the Management > Index Patterns section of Kibana do you see your expected geo field as a geo_point type?

Example:

Regards,
Lee


(Frank Trezza) #3

I see it as NUMBER type - how do I change that?


(Frank Trezza) #4


(Lee Drengenberg) #5

Hi Frank,
I'm going to refer you to some logstash posts like;

You can also search that logstash forum for geopoint to find other posts about it.

If you're still having problems, please post another question there. Also, please include the logstash version you're using as things do change from release to release.

Regards,
Lee


(Lee Drengenberg) #6

Hi Frank,
I chatted with a Logstash expert and he said, you would need to delete your index, then either name your field (target) to geoip.location (which logstash maps to a geopoint), or add the mapping for your dstip_geoip to be a geopoint.

Lee


(Frank Trezza) #7

How do I add the mapping for dstip_geoip to be a geopoint?

I assume I have to modify this code # Parse GeoIP info

if "dstip-private" not in [tags] {
geoip {
source => "dstip"
target => "dstip_geoip"
fields => ["country_name", "region_name", "city_name", "location"]
}
}
}


(Anh) #8

Which ES version are you using?
If you are on ES 2.x, you can install elasticsearch-kopf plugin to add/manage index mapping templates. For 5.x, you can use Developer Tool/Sense on Kibana


(Frank Trezza) #9

I am using 5.x, how do you use "sense" I have developer tools installed


(Anh) #10

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html


(Frank Trezza) #11

I've reviewed the document, and I don't mind rebuilding my index - but I'm still unsure how to rename the field


(Frank Trezza) #12

Someone please help - I really want to get this working. I've installed developer tools and I know I have to make a curl request with something like this

PUT /attractions
{
"mappings": {
"restaurant": {
"properties": {
"name": {
"type": "string"
},
"location": {
"type": "geo_point"
}
}
}
}
}

but don't know exactly what to put in it.


(Anh) #13

This is how I process IIS log and add GeoIP data http://www.secureict.info/2016/07/elastic-stack-process-iis-logs.html.


(Frank Trezza) #14

thank you anhlqn


(Frank Trezza) #15

this is why my current index looks like


(Anh) #16

First of all, are you familiar with using Dev tool to manage index templates? There are a few things you should know:

  • Create a new index template
  • Update existing index template
  • Update mappings for existing indexes (add more field mappings actually, you can't change existing mappings on indexes AFAIK)

On your current indexes a few things that I can tell:

I see 3 data types: "8", "12", and "sonicwall". It could be caused by wrong grok patterns. Don't send data from Logstash to Elasticsearch until you are sure the data are correctly formatted/transformed as you want. Use either stdout or sending out to local file to test and tune the LS config

Your dstip_geoip.location still has wrong data type

 "dstip_geoip": {
            "properties": {
              "city_name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "country_name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "location": {
                "type": "float"
              },
              "region_name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }

A few things you can do to get this work.

  1. Use stdout in LS config and see why data has "type": "8" and "type": "12". All data should have "type": "sonicwall" so that your filters can pick up all messages and process correctly
  2. Delete existing indexes if you can. If not, use a different index pattern to start fresh. Something like sonicwall-%{+YYYY.MM.DD} will do. Sample LS config
  3. Add this mapping template to your ES cluster using Dev tool. Data types should be updated to match v5.x since this mapping template is for 2.x
  4. Send data into ES and see if the dstip_geoip.location field is mapped correctly as geo_point.

(Frank Trezza) #17

On step 3-
how do I find the 2.x data type equivalents in 5.x - I looked at https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html but all the "type" sections seem to be listed there that are in the linked 2.0 template


(Anh) #18

The main difference is string type. Text and keyword now replace string type.

2.x mapping

"dstname": {
          "include_in_all": true,
          "index": "not_analyzed",
          "type": "string"
        },

5.x mapping

"dstname": {
          "include_in_all": true,
          "type": "keyword"
        },

I haven't used 5.x a lot so I don't know all the breaking changes in 5.x


(system) #19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.