Copy complete event to a field of a new event

Elastic Stack Logstash
1

Hi,

I want to copy all fields of an event to a field of a new event in a logstash filter

This

{
        " agent ": " Mozilla / 5.0(compatible; MSIE 9.0)",
        " ip ": " 192.168.24.44 ",
        " request ": " / index.html "
        " response ": {
            " status ": 200,
            " bytes ": 52353
        },
        " ua ": {
            " os ": " Windows 7 "
        }
}

should become

{
    "original_event" : {
        " agent ": " Mozilla / 5.0(compatible; MSIE 9.0)",
        " ip ": " 192.168.24.44 ",
        " request ": " / index.html "
        " response ": {
            " status ": 200,
            " bytes ": 52353
        },
        " ua ": {
            " os ": " Windows 7 "
        }
    }
}

Any smart hints?

2

This is not smart but works.

filter {
  ruby {
    code => "
    event.to_hash.each{|k,v|
      if (!k.start_with?('@'))
        event.set('[original_message]['+k+']', v)
        event.remove(k)
      end
    }"
  }
}

And if (possibly only if) you are using Elasticsearch input plugin, there are target parameter for that purpose.

3

I was thinking about

    new_event = LogStash::Event.new
    event.to_hash.each{|k,v|
      if (!k.start_with?('@'))
        new_event.set('[original_message]['+k+']', v)
      end
      return [new_event]
    }"

But I am not 100% sure that the return is correct.

Anyone knows?

Regards Hans

1 Like
4

I have no idea what is going wrong
If I implement this:

ruby {
    code => '
        # Move everything to data
        event.set("data", event.to_hash)

        # Remove fields other than data
        fields = event.to_hash.keys
        fields.each{|field|
            if (field != "data")
                event.remove(field)
            end
        }
    '
}

The entire event is empty.
As if a event.cancel has been performed

Any clues? Heeeellp :woozy_face:

5

Your script looks working fine in my environment (Windows, Logstash 7.16.3).. What causes the difference??

input {
  stdin{}
}
filter {
  ruby {
    code => '
        # Move everything to data
        event.set("data", event.to_hash)

        # Remove fields other than data
        fields = event.to_hash.keys
        fields.each{|field|
            if (field != "data")
                event.remove(field)
            end
        }
    '
  }
}
output {
  stdout{
    codec=>rubydebug{ metadata => true }
  }
}

>>test

{
    "data" => {
        "@timestamp" => 2022-01-28T15:30:16.269Z,
              "host" => "<my host>",
          "@version" => "1",
           "message" => "test\r"
    }
}
6

I have no idea.
But I found this:

Looks very much the same.

If have created a script and pointed have the path variable point to it.

In this script I created a new event with

def filter(event)

docs = []
new_event = LogStash::Event.new

*<code>*

docs.push(new_event)

return docs

and fill this event with correct format

To no avail

I cannot remove the fields in a hardcoded way because the formats of the events differ.

Regards Hans

7

will try with 7.16 too

8

Hello
I found out that as soon as I do :
event.remove("tags")
The entire contents of the event are gone.

Regards Hans-Peter

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.