Error when copying a field to multiple others in a single mutate block

Elastic Stack Logstash
1

I noticed today, that copying the same field to multiple destinations inside the same mutate block, leads to a _mutate_error. Splitting the copy operation up to two mutate blocks works without any problem. Is this an intended behavior or should this be considered a bug?

The following logstash.yml leads to a _mutate_error tag.

    input {
        generator {
            lines => [
                'May 28 11:11:01 threatarmor ta:BLOCKED_CONNECTION: UTC_TIME_MS="1590657056547" LOCAL_IP="1.1.1.1" LOCAL_PORT="7047" REMOTE_IP="2.2.2.2" REMOTE_PORT="53" PROTOCOL="UDP" INITIATOR="LOCAL" INTERFACE="1b" REASON="IP_BLOCK" REASON_INFO_IP="2.2.2.2/32"'
            ]
        }
    }

    filter {
        grok {
            match => { "[message]" => "%{SYSLOGTIMESTAMP:syslogtimestamp} threatarmor ta:%{DATA:threatarmor.action}: %{GREEDYDATA:threatarmor.kv}"}
        }
         kv {
            field_split => " "
            value_split => "="
            source => "threatarmor.kv"
            target => "threatarmor"
            transform_key => "lowercase"
            remove_field => ["threatarmor.kv"]
        }
    }

    filter {
        mutate {
            copy => { "[threatarmor][remote_ip]" => "[source][address]" }
            copy => { "[threatarmor][remote_ip]" => "[source][ip]" }
        }
    }

    output { stdout {} }

This config works:

    input {
        generator {
            lines => [
                'May 28 11:11:01 threatarmor ta:BLOCKED_CONNECTION: UTC_TIME_MS="1590657056547" LOCAL_IP="1.1.1.1" LOCAL_PORT="7047" REMOTE_IP="2.2.2.2" REMOTE_PORT="53" PROTOCOL="UDP" INITIATOR="LOCAL" INTERFACE="1b" REASON="IP_BLOCK" REASON_INFO_IP="2.2.2.2/32"'
            ]
        }
    }

    filter {
        grok {
            match => { "[message]" => "%{SYSLOGTIMESTAMP:syslogtimestamp} threatarmor ta:%{DATA:threatarmor.action}: %{GREEDYDATA:threatarmor.kv}"}
        }
         kv {
            field_split => " "
            value_split => "="
            source => "threatarmor.kv"
            target => "threatarmor"
            transform_key => "lowercase"
            remove_field => ["threatarmor.kv"]
        }
    }

    filter {
        mutate {
            copy => { "[threatarmor][remote_ip]" => "[source][address]" }
        }
        mutate {
            copy => { "[threatarmor][remote_ip]" => "[source][ip]" }
        }
    }

    output { stdout {} }
2

Yes, it produces the error

Exception caught while applying mutate filter {:exception=>"Cannot cast org.jruby.specialized.RubyArrayTwoObject to org.jruby.RubyString"

The problem is that there is only one @copy object in the mutate class, so when the configuration is read it has to merge all the copy options into it. I try to avoid using the same option more than once, so normally I would write that as

mutate {
    copy => {
        "[threatarmor][remote_ip]" => "[source][address]"
        "[threatarmor][remote_ip]" => "[source][ip]"
    }
}

but of course that is not going to do what you want because you cannot have two fields with the same name in a hash.

I doubt the underlying bug in the configuration parser will ever get addressed, so your only option is two filters.

2 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.