Getting key value from nested "event" element

Elastic Stack Logstash
1

This one has me stumped, and I wonder if there's a subtlety I'm missing.

I have an (unfortunately structured) log event that records an 'assessment' attempt in our learning management system. It looks something like this:

{
  "foo": "bar",
  ...some more top level key/values...
  "event": {
     "submitted_answer": {
        "Q1": {
            ... details on Q1
       }
    }
}

I want to read in the first key string "Q1" and assign it as a value somewhere so I can use it more easily in Kibana. That Q1 is really the ID of an assessment so I'd much rather have something like "question_id":"Q1" at the top level of the document.

However, each method I've tried to grab that "Q1" value (or whatever value it is, e.g. "Q2","Q3") doesn't seem to work.

My most recent try looks like this:

filter {
      ruby {
            code => "
                sa = event.get([event][submitted_answer])
                event.set('question_id', sa.keys[0])
            "
        }
}

Which gives me the error...

Ruby exception occurred: undefined local variable or method `submitted_answer' for #<LogStash::Filters::Ruby:0x4236042c>

I wonder if the internal key string "event" is somehow confusing the event api?

As an aside, I wanted to rename that "event" key to something more sane -- I thought I should do something like this:

mutate {
    rename => { "[event]" => "pb_event"}
}

But I actually have to do something like this:

mutate {
            rename => { "[event][event]" => "pb_event"}
        }

So then trying to apply that lesson to my ruby code, like this:

ruby {
            code => "
                sa = event.get([event][event][submitted_answer])
                event.set('question_id', sa.keys[0])
            "
        }

Gives me a different error:

Ruby exception occurred: can't convert LogStash::Event into Integer
2 
           sa = event.get([event][submitted_answer])

You need to quote the field name:

            sa = event.get('[event][submitted_answer]')
3

Aha! Thanks for the help, Magnus.

However, it turns out I have to reference the top level event in the argument, like:

 ruby {
            code => "
                sa = event.get('[event][event][submitted_answer]')
                event.set('question_id', sa.keys[0])
            "
        }

Which is interesting as it doesn't seem consistent: I would have thought by calling a method on "event" I wouldn't have to refer to the parent "event" it in the string.

If I get some other value I don't think I have to...e.g. if I get 'foo' from my example above, it's event.get('[foo]') not event.get('[event][foo]')

4

Which is interesting as it doesn't seem consistent: I would have thought by calling a method on "event" I wouldn't have to refer to the parent "event" it in the string.

That doesn't make any sense. What does a raw event look like? Use a stdout { codec => rubydebug } output.

5

Yes! You're right. It doesn't make sense.

It turns out an Open edX component was sending me some events where an [event] child property was incorrectly duplicated at [event][event]...but only sometimes, which is why I didn't see it in the log entries I checked and therefore thought it was due to my Logstash code.

Thanks once again.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.