Geo_Point and Logstash

[SteveSmith]

Hi Everyone,

I am having a hard time lately understanding how the Geo_Point type is created. I found that when I create the pipeline (using this link: https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-geoip.html) and ship Packetbeat directly to the cluster I get the Geo_Point data type. But, if i run Packetbeat though Logstash, well that data type is never created. From my limited understanding, it seems that ES would create this when data is shipped to the pipeline, which confuses me as to why it would matter who does the shipping. Below is the config file i am using for Logstash, any help on this would be wonderful.

input {
    beats {
        port => 5044
    }
}
filter {
    if [agent][type] == "packetbeat" {
        if ("xxx.xxx" in [client][ip] or "xxx.xxx" in [client][ip]) and ("xxx.xxx" in [server][ip] or "xxx.xxx" in [server][ip]) {
            mutate {
                add_field => {"traffic_direction" => "Internal"}
            }
        }
        else if "xxx.xxx" in [server][ip] or "xxx.xxx" in [server][ip] {
            mutate {
                add_field => {"traffic_direction" => "Inbound"}
            }
        }
        else if "xxx.xxx" in [client][ip] or "xxx.xxx" in [client][ip] {
            mutate {
                add_field => {"traffic_direction" => "Outbound"}
            }
        }
        else {
            mutate {
                add_field => {"traffic_direction" => "Unkonwn"}
            }
        }
    }
}
output {
    elasticsearch {
        hosts => "https://XXXXXXXXXXXX.us-east-1.aws.found.io:9243"
        user => "elastic"
        password => "xxxxxxx"
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        pipeline => "geoip-info"
    }
}

[Badger]

Are you aware that there is a cidr filter that can do network matching?

It is not clear to me that client.ip in the pipeline configuration refers to what we would reference as [client][ip] in logstash. It might be a field with a period in the name.

[SteveSmith]

I didn't know that there was a cidr filter for logstash, might be helpful for my if statements. But, would that cause the the problem with the geo_point? I have tested in the past without the filter section in the config and still had the same problem with the geo_point.

[Badger]

@SteveSmith

would that cause the the problem with the geo_point?

No. As I said, I suspect a mismatch of the field names.

You could do the enrichment using a geoip filter rather than a pipeline.

[SteveSmith]

Would that be the preferred way? I am still a bit new to things in the stack and I really only had an issue with the geo enrichment part of this. Those filters work for the most part but they are not the issue i have had. It was mainly losing that geo_point info when pushing Packetbeat though Logstash compared to having Packetbeat send direct to ES.

[Badger]

@SteveSmith

Would that be the preferred way?

Whatever works is the preferred way to me