Copy complete event to a field of a new event

[HansPeterSloot]

Hi,

I want to copy all fields of an event to a field of a new event in a logstash filter

This

{
        " agent ": " Mozilla / 5.0(compatible; MSIE 9.0)",
        " ip ": " 192.168.24.44 ",
        " request ": " / index.html "
        " response ": {
            " status ": 200,
            " bytes ": 52353
        },
        " ua ": {
            " os ": " Windows 7 "
        }
}

should become

{
    "original_event" : {
        " agent ": " Mozilla / 5.0(compatible; MSIE 9.0)",
        " ip ": " 192.168.24.44 ",
        " request ": " / index.html "
        " response ": {
            " status ": 200,
            " bytes ": 52353
        },
        " ua ": {
            " os ": " Windows 7 "
        }
    }
}

Any smart hints?

[Tomo_M (Tomohiro Mitani)]

This is not smart but works.

filter {
  ruby {
    code => "
    event.to_hash.each{|k,v|
      if (!k.start_with?('@'))
        event.set('[original_message]['+k+']', v)
        event.remove(k)
      end
    }"
  }
}

And if (possibly only if) you are using Elasticsearch input plugin, there are target parameter for that purpose.

[HansPeterSloot]

I was thinking about

    new_event = LogStash::Event.new
    event.to_hash.each{|k,v|
      if (!k.start_with?('@'))
        new_event.set('[original_message]['+k+']', v)
      end
      return [new_event]
    }"

But I am not 100% sure that the return is correct.

Anyone knows?

Regards Hans

[HansPeterSloot]

I have no idea what is going wrong
If I implement this:

ruby {
    code => '
        # Move everything to data
        event.set("data", event.to_hash)

        # Remove fields other than data
        fields = event.to_hash.keys
        fields.each{|field|
            if (field != "data")
                event.remove(field)
            end
        }
    '
}

The entire event is empty.
As if a event.cancel has been performed

Any clues? Heeeellp

[Tomo_M (Tomohiro Mitani)]

Your script looks working fine in my environment (Windows, Logstash 7.16.3).. What causes the difference??

input {
  stdin{}
}
filter {
  ruby {
    code => '
        # Move everything to data
        event.set("data", event.to_hash)

        # Remove fields other than data
        fields = event.to_hash.keys
        fields.each{|field|
            if (field != "data")
                event.remove(field)
            end
        }
    '
  }
}
output {
  stdout{
    codec=>rubydebug{ metadata => true }
  }
}

>>test

{
    "data" => {
        "@timestamp" => 2022-01-28T15:30:16.269Z,
              "host" => "<my host>",
          "@version" => "1",
           "message" => "test\r"
    }
}

[HansPeterSloot]

I have no idea.
But I found this:

Ruby Filter - Event.Remove Inconsistent Results

Problem Statement I believe there is a bug in the ruby filter plugin when using event.remove Expected Output: ... event_data.sql_text => "some sql text" event_data.logType => "Oracle" ... With no root fields (e.g., sql_text or logType). All data should reside under event_data nested object. Background: I am using docker instances to run an ES stack (3 Elasticsearches, 1 Logstash). I have tested this with docker versions 6.0.0 and 6.3.0 for Logstash. I'm pulling in data from variety of source…

Looks very much the same.

If have created a script and pointed have the path variable point to it.

In this script I created a new event with

def filter(event)

docs = []
new_event = LogStash::Event.new

*<code>*

docs.push(new_event)

return docs

and fill this event with correct format

To no avail

I cannot remove the fields in a hardcoded way because the formats of the events differ.

Regards Hans

[HansPeterSloot]

will try with 7.16 too

[HansPeterSloot]

Hello
I found out that as soon as I do :
event.remove("tags")
The entire contents of the event are gone.

Regards Hans-Peter