Working with JSON array of objects via Ruby code block

Elastic Stack Logstash
1

I have a JSON array of nested objects. I wrote a filter

json { 
  source => ['body']
  target => "parsed"
}

ruby {
  code => "mapped = event.get('parsed').map{
    |h| [h['k1'], h['k2'],h['k3'],h['k4']]
  }; event.set('mapped', mapped)"
}

I have two questions

  1. there is another key, say h['k5']. This key contains a json itself. I can access h['k5'] but I cannot access it's nested objects. For example I try h['k5']['l1'] for one of it's keys and that does not work. What do I need to do to access it's fields from the ruby code?

<Resolved, was using bad keys. My bad. #2 is still an open question.>

  1. Currently in above I am returning an array. If I wish to return a json Object instead of an Array, how do I do that?
2

What does that mean, and what does your data actually look like once it is parsed. Can you show an event from

output { stdout { codec => rubydebug } }
3

Regarding 1) I resolved my issue, I was using bad keys. So 1) does work.

#2 is still an open issue. Instead of returning an array using the map function, what is the appropriate way to return a json object using the map function body?

4

You have not explained what you want the result to look like. There are no JSON objects in logstash. You wrote your code so that .map returns an array. You could have returned something else.

5

I'm specifically asking what is a good approach for returning a valid JSON object. I could take my key values and form JSON from strings. For example if h[k1] .. h[kn] all return strings, I could try writing JSON using String interpolation. Example to add a key for each value extracted.

But that seems rather crude and messy. Is there an alternative?

Can I use the JSON module from ruby's stdlib and JSON.generate(...) on a ruby hash?

6

The ask does not make sense to me. There are no JSON objects in a logstash pipeline. Fields can be objects that contain other objects, but that does not make them JSON.

Do you want serialized JSON in a field? You could install the json_encode filter for that.

7

Do you want serialized JSON in a field?

Doesn't my question of JSON.generate(...) imply that I am trying to serialize JSON?

You could install the json_encode filter for that.

Can I use json_encode by surrounding the ruby code block? Then do I need to nest the ruby / code block inside the json_encode? I assume I would need to escape the json_encode "arguments" if that were the case , which sounds difficult.

Let me try make the problem clear to you. I have the following code. I am currently outputting a ruby array .

json { 
  source => ['body']
  target => "parsed"
}

ruby {
  code => "mapped = event.get('parsed').map{
    |h| [h['k1'], h['k2'],h['k3'],h['k4']]
  }; event.set('mapped', mapped)"
}

I prefer to output serialized JSON (serialized JSON is encoded objects in a string).

Requirements ->

Instead of returning an array in the map function, return the serialized Json string. For each key 'k1' , 'kn' use a value key1..keyN for the corresponding JSON object.

Example {"key1":<h['k1']>, ..., "keyN":<h['kN']>} where <h['k1']> is the value extracted by the map function.

  1. create a structured serialized JSON (serialized JSON is encoded objects in a string)
  2. do so from within a ruby code block
8

You are not making it clear. Instead of showing what the input data looks like and exactly what result you want, you are showing us examples of operations upon unseen data that do not produce what you want.

9

Input data to the ruby is a JSON array of objects as explained.

I'll figure it out on my own and post the solution.

10

Can I use the JSON module from ruby's stdlib and JSON.generate(...) on a ruby hash?

To answer my own question: It took me awhile to lock down the syntax. But I was able to use init and JSON.generate along with the map function to create a JSON string via the Ruby code block.

init => "require 'JSON'"

However one possibly very tricky problem remains.

My output now looks like

"body" : [
"{\"key1\" : \"value1\",\"keyn\" : \"valuen\"}",
"{\"key1\" : \"value1\",\"keyn\" : \"valuen\"}",
"{\"key1\" : \"value1\",\"keyn\" : \"valuen\"}"
]

That is each JSON object is wrapped inside a String in escaped syntax. I think it might be very difficult to extract each object from it's String wrapper because claims that LS needed Ruby to work with Arrays in the first place. However I will see if I can simply apply another filter to extract the String. I hope I can extract the String and complete this work quickly.

11

I was able to solve my problem. It was not necessary to use the JSON module. Returning a Ruby Hash and the output from Logstash seems to be a valid JSON object.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.