Get rid of .keyword

[svenvg93]

Hi,

I'm still a ELK noob so bear with me here ;).

I've setup Logstash to send the output of fping command to Elasticsearch and add some geoip data to it.

Fping output;

ec2.eu-west-1.amazonaws.com  : xmt/rcv/%loss = 30/29/3%, min/avg/max = 27.1/27.4/28.0

I already remove some unwanted fields and converted some values. The thing I cannot seems to figure out is why I get same double fields of which one is name fields.keyword

@timestamp:
Jun 25, 2021 @ 23:18:35.680
@version:
1
avg:
89.5
geoip.as_org:
Amazon.com, Inc.
geoip.as_org.keyword:
Amazon.com, Inc.
geoip.asn:
16,509
geoip.country_code2:
US
geoip.country_code2.keyword:
US
geoip.country_name:
United States
geoip.country_name.keyword:
United States
geoip.ip:
54.239.28.168
loss:
0
max:
90.9
min:
89
tags:
fping
tags.keyword:
fping
target:
ec2.us-east-1.amazonaws.com
target.keyword:
ec2.us-east-1.amazonaws.com
type:
fping
type.keyword:
fping
_id:
wScJRXoBnY3E-uxC7RYW
_index:
logstash-2021.06.25-000001
_score:
-
_type:
_doc

How can I remove the .keyword fields since they are double.

full Logstash config;

input {
    exec {
        command => "/usr/bin/fping -q -c 30 -B1 -r1 < /etc/logstash/fping.conf 2>&1"
        interval => 10
        type => "fping"
        tags => [ "fping" ]
    }
}
filter {
    if "fping" in [tags] {
        split {
        }
        grok {
            match => { "message" => "%{IPORHOST:target}.*loss = %{DATA}\/%{DATA}\/%{NUMBER:loss}%{DATA}= %{NUMBER:min}\/%{NUMBER:avg}\/%{NUMBER:max}" }
        }
        if [reponse] == "-" {
            drop { }
        } else {
            mutate {
                convert => { "avg" => "float" }
                convert => { "loss" => "float" }
                convert => { "max" => "float" }
                convert => { "min" => "float" }   
                remove_field => "[command]"       
                remove_field => "[message]"  
                remove_field => "[host]"                                   
            }
        }
        geoip {
            source => "target"
            database => "usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-Country.mmdb"
            remove_field => "[geoip][continent_name]"
        }
        geoip {
            source => "target"
            database => "usr/share/elasticsearch/modules/ingest-geoip/GeoLite2-ASN.mmdb"
            remove_field => "[geoip][as_org][keyword]"
        }
    }
}
output {
    elasticsearch {
        hosts    => [ "localhost" ]
        index => "fping-%{+YYYY.MM.dd}"
    }
}

Any feedback on it is appreciated

Thnx!

[Badger]

The rationale for .keyword is explained in this blog post. Once you understand why they exist, if you want to stop them being created by default you will have to create your own template to replace the default.