Trouble with conditionals in aggregate filter

Elastic Stack Logstash
1

I am working to pull my data into elasticsearch using logstash and running into some trouble. I'll do my best to explain the situation:

I'm pulling data via jdbc and everything seems to be working with the exception of one issue.

my data looks something similar to:
id, name, test_name, test_score, question_name, question_score

  map['tests'] ||= []
  map['tests'] << {event.get('test_name') => event.get('test_score')}
  map['assays'] ||= []
  map['assays'] << {event.get('question_name') => event.get('question_score')}

Now, as expected, if I have a test name of "Final" with multiple questions "Bonus", "Essay" etc I will have multiple entries in the output for tests. I am trying to use a conditional to filter this but if I do:
if !map['tests'][event.get('test_name')]

I get No implicit conversion of string into integer

if I do:
if event.get('test_name') not in map['tests']

I get unexpected keyword_not, if I reverse the logic I get unexpected keyword_in.

I found a page on SO mentioning using something more similar to
if ! map['tests'][0][event.get('test_name')]

This results in undefined method '[]' for nil:NilClass

I'm really unsure of where to go from here. If anyone could point me in the right direction I'd greatly appreciate it!

2

I do not understand what you are trying to test here. map['tests'] is an array of hashes. Are you trying to test whether any of the hashes in the array have a key equal to the value of the [test_name] field.

3

that's exactly it,

There might be multiple fields that end up as map['tests']['Final'] for example, they will be exactly the same so I'm hoping to filter them out. For example I might currently end up with something akin to:

tests{
    "Final":86
    "Final":86
    "Final":86
    "Midterm":70
}

I can't easily handle this at the SQL side because the second aggregated field ("assays" in the above example) will have multiple entries associated with a test.

The essential layout in the database is that client has a one to many relationship with tests which then has a one to many relationship with assays.

4

That's not what you are building. You are building an array of hashes.

[{"Final":"86"}, {"Final":"86"}, {"Final":"86"}, {"Midterm":"90"}]
5

That's fair, I'm not very familiar with what is happening. My example was more meant to be informal on what data exists and your example is more similar to what I see in kibana.

My questions is, how would I avoid having 4 hashes in the array and only end with the two unique ones[{"Final":"86"}, {"Midterm":"90"}] ? Is this even possible?

If there's a better way to import the data in a different format I'm open to it. I'm just getting started with elasticsearch and logstash (clearly) and don't currently have any hard constraints on how I organize my data since this project is currently only going to be used by me.

6

Then I think you want the tests object to be a hash.

map["tests"] ||= {}
map["tests"][ event.get("test_name") ] = event.get("test_score")
7

This worked beautifully. Thank you so much for your help!