I would like to access nested data from an array

Elastic Stack Logstash
1

I have a data structure looking like this:

printed as codec => rubydebug
{
"sequence_number" => 557,
"event_tags" => [
[0] {
"value" => "nacl-st-qkrrfq-ppu-server-6y35vxntjont",
"key" => "originhost"
}
]
.
.
.
.
}

or printed as codec => line
"event_tags":[{"value":"nacl-st-alcags-ppu-server-zwul4nphc2lc","key":"originhost"}]

or from kibana:
event_tags {
"value": "nacl-st-ibgki7-ppu-server-bl46vcsakx5f",
"key": "originhost"
}

I would like ta add a field calld "originhost" which has the value "nacl-st-alcags-ppu-server-zwul4nphc2lc" in this example.

I have tried the following in a mutate filter:
add_field => {"originhost" => "%{[0][event_tags][key]}"}
and
add_field => {"originhost" => "%{[event_tags][key]}"}

but it does not work.

The second one causes java errors in logstash log and the first one "originhost = %{[0][event_tags][key]}"

How should I write the filter?

Br Mathias

2

if your event structure is:

{
  "sequence_number" => 557,
  "event_tags" => [
    [0] {
      "value" => "nacl-st-qkrrfq-ppu-server-6y35vxntjont",
      "key" => "originhost"
    }
  ]
}

And you want to extract the array element where the key is "originhost" then you need the ruby filter.
This is because there's no way of knowing if there is more than on element with originhost (or none)
With ruby filter, you can do:

filter { ruby { code => "event.get('event_tags').find {|h| h['key'] == 'originhost'}['value']" } }
3

Hi Juao,

Thanks for your reply.
I still have problems to make the ruby filter have any affect.
I do not see any new string field called originhost.

My filter config looks like this:

if "logstash-eventbus-tcp" in [tags] {
mutate {
remove_tag => [ "logstash-eventbus-tcp" ]
rename => { "type" => "eventtype" }
add_field => { "type" => "event/%{[eventtype]}" }
}

mutate {
rename => { "tags" => "event_tags" }
add_field => { "originhostX" => "%{[event_tags]}" }
}

ruby { code => "event.get('event_tags').find {|h| h['key'] == 'originhost'}['value']" }
}

The kibana json format of an event looks like this:
{
"_index": "events-2017.01.17",
"_type": "event/system/counter/pbuf",
"_id": "AVmsB530vNaxVEhbQd0d",
"_score": null,
"_source": {
"sequence_number": 881,
"event_tags": [
{
"value": "nacl-st-qm8jef-ppu-server-sy4b5tjkl3iz",
"key": "originhost"
}
],
"@timestamp": "2017-01-17T10:44:57.666Z",
"originhostX": "{value=nacl-st-qm8jef-ppu-server-sy4b5tjkl3iz, key=originhost}",
"data": {
"resources": [
{
"identity": "1",
"type": "ue_bearer_id"
},
{
"identity": "bearer_deleted",
"type": "name"
},
{
"identity": "2017-01-17 11:44:57.617778",
"type": "time"
}
]
},
"port": 59873,
"@version": "1",
"host": "10.68.32.209",
"producer": "6df18e784e60412830f676ed811bc6d5c8721e82",
"eventtype": "system/counter/pbuf",
"type": "event/system/counter/pbuf",
"timestamp": "2017-01-17T10:44:57.937752962Z"
},
"fields": {
"timestamp": [
1484649897937
],
"@timestamp": [
1484649897666
]
},
"sort": [
1484649897666
]
}

4

there is one thing missing from that code, which is setting the retrieved value back into the event:

so while this code has no side effect because it only retrieves the data:

ruby { code => "event.get('event_tags').find {|h| h['key'] == 'originhost'}['value']" }

to the value back into the event you need

ruby { code => "value = event.get('event_tags').find {|h| h['key'] == 'originhost'}['value']; event.set('originhost', value)" }
5

Many thanks to you Joao.

It works perfectly!

Must ask you right away. Is it possible to make a generic filter which creates fields of each key with it's corresponding value without having to specify each key?
For example from the following data structure:
{
"_index": "events-2017.01.17",
"_type": "event/system/counter/pbuf",
"_id": "AVmsadkeP1_tDGOJZ1RF",
"_score": null,
"_source": {
"data": {
"counters": [
{
"name": "n_lost_packets",
"value": 0
},
{
"name": "n_sent_bytes",
"value": 130852808
},
{
"name": "n_received_packets",
"value": 51705
},
{
"name": "n_sent_packets",
"value": 94488
},
{
"name": "n_ce_bytes",
"value": 0
},
{
"name": "n_received_bytes",
"value": 2883215
}
],

I would like to see 6 fields with the names n_lost_packets, n_sent_bytes, n_received_packets, n_sent_packets, n_ce_bytes and n_received_bytes.

What would the find and set command look like for this task?

Thanks
Mathias

6

I think the major issue here is that the source application is sending essentially a hash in array form.
If the counters element was:

"counters": {
  "n_lost_packets": 0,
  "n_sent_bytes": 130852808
}

instead of:

"counters": [
  {
    "name": "n_lost_packets",
    "value": 0
  },
  {
    "name": "n_sent_bytes",
    "value": 130852808
  }
]

it would be simpler. If you want to programatically you need ruby filter since, in your case each hash contains a key under "key" and a value under "value", but it could be something else, also there can be duplicates, some elements may only contain key or value (errors happen), etc

in your case, to iterate over counters and set each property on event, you can do:

filter { ruby { code => "event.get('counters').each {|hash| event.set(hash['key'] = hash['value']) }" } }

(this code wasn't tested)

7

Thanks Joao,

From the latest filter I get an ruby exception:
[2017-01-18T06:50:23,571][ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `each' for nil:NilClas

In Kibana I can see that there is an exception tag called "_rubyexception" added.

{
  "_index": "events-2017.01.18",
  "_type": "event/system/counter/pbuf",
  "_id": "AVmwVgFvpR5aTmxaCXPP",
  "_score": null,
  "_source": {
    "sequence_number": 568,
    "event_tags": [],
    "@timestamp": "2017-01-18T06:49:01.895Z",
    "data": {
      "counters": [
        {
          "name": "n_lost_packets",
          "value": 0
        },
        {
          "name": "n_sent_bytes",
          "value": 474496416
        },
        {
          "name": "n_received_packets",
          "value": 176983
        },
        {
          "name": "n_sent_packets",
          "value": 339238
        },
        {
          "name": "n_ce_bytes",
          "value": 0
        },
        {
          "name": "n_received_bytes",
          "value": 9745601
        }
      ],
      "resources": [
        {
          "identity": "bearer_path_pm",
          "type": "name"
        },
        {
          "identity": "1",
          "type": "ue_bearer_id"
        },
        {
          "identity": "",
          "type": "pm_id"
        },
        {
          "identity": "3",
          "type": "path_identifier"
        },
        {
          "identity": "2017-01-18 07:49:02.267653",
          "type": "time"
        }
      ]
    },
    "port": 37438,
    "@version": "1",
    "host": "10.68.32.232",
    "producer": "ddc3eb8dfc39e8b785983a0cc57848c5621d1e08",
    "eventtype": "system/counter/pbuf",
    "type": "event/system/counter/pbuf",
    "timestamp": "2017-01-18T06:49:03.016452074Z",
    "tags": [
      "_rubyexception"
    ]
  },
  "fields": {
    "timestamp": [
      1484722143016
    ],
    "@timestamp": [
      1484722141895
    ]
  },
  "sort": [
    1484722141895
  ]
}

I have tried to check if the event is nil or not and that prevents the exception.
ruby { code => "myArr = event.get('data.counters')
if myArr
event.get('data.counters').each {|hash| event.set(hash['key'] = hash['value']) }
end" }

I have tried both myArr = event.get('counters') and myArr = event.get('data.counters'). It gives the same result. No action.

Any ideas what is wrong?

Kind regards
Mathias

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.