Using JSON filter to expand the field created by terms aggregation in transform

Elastic Stack Kibana
,
1

I am using transforms to aggregate a field event by terms so I get a count for each event for all sessionIds. This seems to be working fine and I get a JSON string as a new field events. But i can't query for the individual terms inside this field, so I'm using an ingest pipeline with the JSON filter to expand the events field. But this is returning the following error -

[transform4] transform has failed; experienced: [Failed to index documents into destination index due to permanent error: [org.elasticsearch.xpack.transform.transforms.BulkIndexingException: Bulk index experienced [110] failures and at least 1 irrecoverable [[1:3] Unexpected character ('i' (code 105)): was expecting double-quote to start field name
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]]. Other failures: 
[IngestProcessorException] message [org.elasticsearch.ingest.IngestProcessorException: org.elasticsearch.xcontent.XContentParseException: [1:3] Unexpected character ('i' (code 105)): was expecting double-quote to start field name
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]]; org.elasticsearch.xcontent.XContentParseException: [1:3] Unexpected character ('i' (code 105)): was expecting double-quote to start field name
 at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 3]]].

It seems to me that the error means that the keys inside the JSON should be enclosed within double quotes, but they already are.
Here is what the generated events field looks like - { "idle": 1, "exited": 1, "text-input-entered": 1, "text-input-changed": 1, "click": 14, "landed": 1 }

2

Transform maps a terms aggregation to a flattened field. An alternative to your ingest pipeline is to create a different mapping for this field. flattened has been chosen, because the terms agg potentially produces a field with lots of different sub-fields. If you don't have a lot of different fields (the default limit is 1000) I suggest to create mappings for them, e.g. map them as long. If you do, you can search and aggregate on these fields. For defining those mappings I suggest to use a dynamic template.

Another solution to this problem has been discussed recently in this thread.

1 Like
3

I'm not really familiar with mappings. Will take some time and go through the documentation on it. Thanks!

Update : I just mapped my events field as an object and it worked like a charm. Thanks a lot again!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.