Trouble with conditionals in aggregate filter

I am working to pull my data into elasticsearch using logstash and running into some trouble. I'll do my best to explain the situation:

I'm pulling data via jdbc and everything seems to be working with the exception of one issue.

my data looks something similar to:
id, name, test_name, test_score, question_name, question_score

  map['tests'] ||= []
  map['tests'] << {event.get('test_name') => event.get('test_score')}
  map['assays'] ||= []
  map['assays'] << {event.get('question_name') => event.get('question_score')}

Now, as expected, if I have a test name of "Final" with multiple questions "Bonus", "Essay" etc I will have multiple entries in the output for tests. I am trying to use a conditional to filter this but if I do:
if !map['tests'][event.get('test_name')]

I get No implicit conversion of string into integer

if I do:
if event.get('test_name') not in map['tests']

I get unexpected keyword_not, if I reverse the logic I get unexpected keyword_in.

I found a page on SO mentioning using something more similar to
if ! map['tests'][0][event.get('test_name')]

This results in undefined method '[]' for nil:NilClass

I'm really unsure of where to go from here. If anyone could point me in the right direction I'd greatly appreciate it!

I do not understand what you are trying to test here. map['tests'] is an array of hashes. Are you trying to test whether any of the hashes in the array have a key equal to the value of the [test_name] field.

1 Like

that's exactly it,

There might be multiple fields that end up as map['tests']['Final'] for example, they will be exactly the same so I'm hoping to filter them out. For example I might currently end up with something akin to:


I can't easily handle this at the SQL side because the second aggregated field ("assays" in the above example) will have multiple entries associated with a test.

The essential layout in the database is that client has a one to many relationship with tests which then has a one to many relationship with assays.

That's not what you are building. You are building an array of hashes.

[{"Final":"86"}, {"Final":"86"}, {"Final":"86"}, {"Midterm":"90"}]
1 Like

That's fair, I'm not very familiar with what is happening. My example was more meant to be informal on what data exists and your example is more similar to what I see in kibana.

My questions is, how would I avoid having 4 hashes in the array and only end with the two unique ones[{"Final":"86"}, {"Midterm":"90"}] ? Is this even possible?

If there's a better way to import the data in a different format I'm open to it. I'm just getting started with elasticsearch and logstash (clearly) and don't currently have any hard constraints on how I organize my data since this project is currently only going to be used by me.

Then I think you want the tests object to be a hash.

map["tests"] ||= {}
map["tests"][ event.get("test_name") ] = event.get("test_score")
1 Like

This worked beautifully. Thank you so much for your help!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.