Push_previous_map_as_event into fields

[mohsin106 (Mohsin)]

Hi,

I'm trying to use push_previous_map_as_event to aggregate and save everything as key/value fields in a document.

My data looks like this:

device: router1
property: mem_prop1 ( there are twelve properties per component-name for a device)
component-name: FPC1:CPU1
value: 1
sequence: 1000 (this appears to be unique per device and component-name)

device: router2
property: mem_prop2
component-name: FPC2:CPU2
value: 2
sequence: 2000

This is my aggregate filter which is storing the property-name and value inside an array:

     if [component-name] and [property-name] and [property-value] {
            aggregate {
                task_id => "%{device}-%{sequence}-%{component-name}"
                timeout_task_id_field => "task_id"
                timeout_timestamp_field => "@timestamp"
                code => "
                    map['sequence'] ||= event.get('sequence')
                    map['device'] ||= event.get('device')
                    map['component-name'] ||= event.get('component-name')
                    map['properties'] ||= []
                    map['properties'] << {event.get('property-name') => event.get('property-value')}
                "
                push_map_as_event_on_timeout => true
                push_previous_map_as_event => true
                timeout => 60
            }
        }
        else {
            drop {}
        }

My output looks like this:

    "properties": [
          {
            "mem-util-packet-dma-utilization": 9
          },
          {
            "mem-util-kernel-flow-table-allocations-failed": 0
          },
          {
            "mem-util-kernel-toe-pkt-transfer-allocations-failed": 0
          },
          {
            "mem-util-kernel-cos-allocations-failed": 0
          },
          {
            "mem-util-kernel-filter-allocations-failed": 0
          },
          {
            "mem-util-kernel-rtt-allocations-failed": 0
          },
          {
            "mem-util-kernel-rt-allocations-failed": 0
          },
          {
            "mem-util-kernel-iff-allocations-failed": 0
          },
          {
            "mem-util-kernel-ifl-allocations-failed": 0
          },
          {
            "mem-util-kernel-size": 3221225472
          },
          {
            "mem-util-kernel-utilization": 26
          },
          {
            "mem-util-kernel-bytes-allocated": 860475368
          }
        ]

I'd like to flatten properties out so that all of the property-names and values get saved as key/value fields in the aggregated document. How do I do that?

Thank you.

[Badger]

@mohsin106

map['properties'] ||= []
map['properties'] << {event.get('property-name') => event.get('property-value')}

Replace that with

map['properties'] ||= {}
map['properties'][event.get('property-name')] = event.get('property-value')

or even

map[event.get('property-name')] = event.get('property-value')

BTW are you sure you want both of these?...

            push_map_as_event_on_timeout => true
            push_previous_map_as_event => true

[mohsin106 (Mohsin)]

@Badger
Thank you. Your suggestion worked. As for your question, I'm now using only

push_previous_map_as_event => true

Do I need to set a timeout (timeout => 60) when using push_previous_map_as_event?

I configured my aggregate filter like this (without timeout) and it's working.

aggregate {
task_id => "%{device}-%{sequence}-%{component-name}"
timeout_task_id_field => "task_id"
timeout_timestamp_field => "@timestamp"
code => "
map['sequence'] ||= event.get('sequence')
map['device'] ||= event.get('device')
map['component-name'] ||= event.get('component-name')
map['properties'] ||= {}
map[event.get('property-name')] = event.get('property-value')
"
push_previous_map_as_event => true
}

I also have another question about the "code =>" syntax. Is that all pure Ruby code between the double quotes or is it some sort of trimmed down version of it? I haven't learned Ruby yet but are there any references that I can read up on that would show me how to utilize Ruby with Elastic Search? I know there is an Event API in ES that we can utilize to get and set events, but just needed more reading material on how we use map['...'] to dynamically create fields and assign values to them. Thank you!

[Badger]

You do not need to set timeout if using push_previous_map_as_event

Yes, it is pure Ruby code in the code option of a ruby filter. I cannot think of a reference to suggest.

[mohsin106 (Mohsin)]

Got it. Thanks again for your help!