Creation of new nested object in Ruby Filter yields no object in ElasticSearch

I've got a logstash pipeline that takes discrete log trace events and writes them to a trace-oriented document in ElasticSearch. Currently I had been storing only the timestamp of the event in an array field in the document. The array is extended via update script as each new trace event comes in. I want to extend this to include the event description so that each timestamp has a description of the activity that was taking place. I'm using the Ruby filter to do this with the following line:

 ruby {    
    code => 'event.set("TraceTimeEntry", {"EntryDescription" => event.get("EventDescription"), "EntryTime" => event.get("TempDateObject")})
			'
  } 

And this line in the Painless update script:

        if (ctx._source.TraceEvents != null) {
		   ctx._source.TraceEvents.add(params.event.get("TraceTimeEntry"));
        } else {
		   ctx._source.TraceEvents = [params.event.get("TraceTimeEntry")];
        }

"TraceEvents" is an array of "TraceTimeEntry" objects.

The "EventDescription" and "TempDateObject" are both accessible in the object and appear in the document as fields in ElasticSearch. Using the rubydebug output there is no error presented but the "TraceTimeEntry" doesn't appear in the transformed output to stdout. "TraceTimeEntry" does, however, appear as a nil (-) field in ElasticSearch. I believe the issue must be some problem with my specification of the nested object as the same code above worked with just the timestamp before I attempted to set the TraceTimeEntry object. Without a warning/error indicator I'm not sure what might be incorrect about this and would appreciate any input.

As a follow up, this is what the resulting array looks like in ElasticSearch where each null represents one of the TraceTimeEntry objects that should be present:

"TraceEvents": [
  null,
  null,
  null,
  null,
  null,
  null,
  null
]

The reason the array was populating with nulls instead of with my data is I didn't realize I was pruning the data I was creating with an intermediate prune filter. Once I added the new field to the whitelist, the array began populating correctly.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.