Change local ip address to "any text" in elasticsearch logs

Elastic Stack Logstash
1

hi all! we collect nginx access logs in elasticsearch.

here is an example of incoming nginx access logs:
kibana-linux.elk.com 10.211.45.45 - [05/Sep/2022:15:26:54 +0300] UNIX-TIME-1662380814.467 "GET /internal/security/me HTTP/1.1" 200 "application/json; charset =utf-8" 353 "https://kibana-linux.elk.com/spaces/space_selector" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 " "sid=Fe26.2**41173fd612c2c0c3ca13f7eeb955e7119efb3d7cfd73a11a29a0ac5c1aedbb24*Zw1pAvrGVFrpl2WwOXz2SQ*_CnHN5GfUYsBvk2qdAbKuB9OXsWfDxPjo9L6QdIIoF6WEcnA6AyjQF89naOGFEyverY_ujS4-MDc0FmEjtriiost-vJQknV47Eh75urCRd8eJwdUTXXtpd1d5XdH5Hqm5ypEPKQF-5cNgRLiZAbRpmY_puuGTGVeG9DtayRqwe3PCWEgaBM4c5CZG7saVIC3JdqpiDBJrsxcb06Yr6O-jWFHMAV7K8kFEk1P0fHDqX7BIdOf6nYEru**206faeae4d13691521f2f15d275923e60d017c6f81972246f335a1433c29c8aa*ZOt5uqlaOxzEI0KI9bDgFLM2Q9ZPgzWF4XZNsxh4z4U" 0.100 "127.0.0.1:5601" NGINX-CACHE-- "5059c3a35ed5704a181a9226a1c3" "-"

in logstash I use the following pipeline and filter (I left only the necessary for an example):

input {
    redis {
        host => "10.206.1.15"
        port => "6379"
        data_type => "list"
        type => "mylog"
        key => "mylog"
        codec => "plain"
    }
}

filter {
    if [type] == "mylog" {
            geoip {
            source => "remote_addr"
            target => "geoip"
            database => "/etc/logstash/GeoLite2-City.mmdb"
        }
        
        mutate {
            add_field => ["[geoip][country_reg_code]", "%{[geoip][geo][country_iso_code]}"]
            gsub => [ "[geoip][country_reg_code]", "%{[geoip][geo][country_iso_code]}", "-" ]
            gsub => [ "[geoip][country_reg_code]", "-%{[geoip][geo][region_code]}", "-" ]
        }
    }
}

output {
    if [type] == "mylog" {
        elasticsearch {
 .......
   }
}
}

using Geoip filter plugin Geoip filter plugin | Logstash Reference [8.4] | Elastic

this works fine with the "global" addresses that are in the GeoLite2-City.mmdb database, but in my logs there are also local IP addresses like 10.211.45.45, 172.17.1.172, etc.

because these addresses are not in the GeoLite2-City.mmdb database, then instead of geoip.country_reg_code, the gsub part from the logstash is supplied, namely "%{[geoip][geo][country_iso_code]}"

image

what condition (expression) in the logstash pipeline do I need to write in order to fulfill the following condition:
if there is no local IP address in the GeoLite2-City.mmdb database, then in this field instead of the gsub part from the logstash, you must specify either the IP address itself, or specify the text "local IP address"

2

You should test if the field [geoip][geo][country_iso_code] exists before your mutate.

Something like this:

if [geoip][geo][country_iso_code] {
    mutate {
        add_field => { "[geoip][country_reg_code]" => "%{[geoip][geo][country_iso_code]}" }
    }
} else {
    mutate {
        add_field => { "[geoip][country_reg_code]" => "local IP address" }
    }
}
3

thanks, I applied your recommendation, but now the field geoip.country_reg_code looks like this

image

4

Did you remove the other mutate?

What is the configuration you are using now?

5

no, didn’t delete the other mutate. need to remove it?

now my pipeline.conf looks like this

input {
    redis {
        host => "10.206.1.15"
        port => "6379"
        data_type => "list"
        type => "mylog"
        key => "mylog"
        codec => "plain"
    }
}

filter {
    if [type] == "mylog" {
            geoip {
            source => "remote_addr"
            target => "geoip"
            database => "/etc/logstash/GeoLite2-City.mmdb"
        }
        
        mutate {
            add_field => ["[geoip][country_reg_code]", "%{[geoip][geo][country_iso_code]}"]
            gsub => [ "[geoip][country_reg_code]", "%{[geoip][geo][country_iso_code]}", "-" ]
            gsub => [ "[geoip][country_reg_code]", "-%{[geoip][geo][region_code]}", "-" ]
        }
    }

if [geoip][geo][country_iso_code] {
    mutate {
        add_field => { "[geoip][country_reg_code]" => "%{[geoip][geo][country_iso_code]}" }
    }
} else {
    mutate {
        add_field => { "[geoip][country_reg_code]" => "local IP address" }
    }
}

}

output {
    if [type] == "mylog" {
        elasticsearch {
 .......
   }
}
}
6

That's the issue then, if you do not remove the other mutate you will keep getting this value since you are not checking if the field exist or not.

You need to have something like this:

filter {
    if [type] == "mylog" {
        geoip {
            source => "remote_addr"
            target => "geoip"
            database => "/etc/logstash/GeoLite2-City.mmdb"
        }
        if [geoip][geo][country_iso_code] {
            mutate {
                add_field => { "[geoip][country_reg_code]" => "%{[geoip][geo][country_iso_code]}" }
            }
        } else {
            mutate {
                add_field => { "[geoip][country_reg_code]" => "local IP address" }
            }
        }
    }
}
1 Like
7

thanks for the help, deleting the first mutate really helped (I was afraid to delete it)

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.