Removing a single value from an array

Elastic Stack Logstash
1

I can't figure out how to remove a single value from an array.

I am trying to build a pipeline where I clone an event do different operation on the copy and finally output an original event into one index and the clone into different index.
In short my pipeline looks as below. I have an input, add @metadata.type to it, then clone this input and change the @metadata.type to different value.
All looks nearly good but after I output both event in the clone my @metadata.type has both the originally added value and the new one.
What do I miss?
How to do it most efficiently? (I don't like current solution anyway)
PIPELINES:

input { stdin { } }
filter {
    json { source => "message" }
    mutate { add_tag => [ "raw" ] }
        mutate {
            add_field => { "[@metadata][type]" => "raw_route" } 
    }  
    # ==== cloning part ====
    clone {
        # Note that the field type will be added to the event by cloning ("type": "modified")
        clones => ['modified']
    }
    if [type] == "modified" {
        mutate {
            remove_field => ["@metadata"]
            remove_field => ["type"]
            add_field => { "[@metadata][type]" => "modified_route" } 
        }
    }
    # ==== modify filter ====
    if [@metadata][type] == "modified_route" {
        mutate { add_field => { "event" => "metrics" } }
    }
}
output { stdout { codec => rubydebug { metadata => true } } }

INPUT:

{"event.type": "reception"}

OUTPUT:

{
    "@timestamp" => 2019-03-12T17:26:08.333Z,
     "@metadata" => {
        "type" => "raw_route"
    },
    "event.type" => "reception",
      "@version" => "1",
       "message" => "{\"event.type\": \"reception\"}",
          "tags" => [
        [0] "raw"
    ]
}
{
    "@timestamp" => 2019-03-12T17:26:08.333Z,
     "@metadata" => {
        "type" => [
            [0] "raw_route",
            [1] "modified_route"
        ]
    },
    "event.type" => "reception",
      "@version" => "1",
       "message" => "{\"event.type\": \"reception\"}",
          "tags" => [
        [0] "raw"
    ]
}

EXPECTED OUTPUT: (Change in @metadata.type in second event, and added "event" => "metrics" to second event)

{
    "@timestamp" => 2019-03-12T17:26:08.333Z,
     "@metadata" => {
        "type" => "raw_route"
    },
    "event.type" => "reception",
      "@version" => "1",
       "message" => "{\"event.type\": \"reception\"}",
          "tags" => [
        [0] "raw"
    ]
}
{
    "@timestamp" => 2019-03-12T17:26:08.333Z,
     "@metadata" => {
        "type" => "modified_route"
    },
    "event.type" => "reception",
      "@version" => "1",
       "message" => "{\"event.type\": \"reception\"}",
         "event" => "metrics",
          "tags" => [
        [0] "raw"
    ]
}
2

add_field is executed before remove_field. You need to use multiple mutate filters to force order.

However, even that will not work because you cannot remove sub-fields of @metadata.

3

Thanks @Badger, could you please advise some working solution?
Is there a way to do it at all?

4

Replace the second mutate+add_field with mutate+replace.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.