Transform documents to create new fields from aggregated data

[Anup_Shirolkar]

Hi,

I am using the opensource version of elastic and want to transform my data as follows:

The metric data is collected from a Hadoop cluster over JMX interface

Mapping:
@queue :keyword
@metric_type :keyword
@cluster :keyword
@value :numeric
@timestamp :timestamp

Data: (in the input index)
{@queue: queue1, @metric_type: used_memory, @value: 1024, @cluster: dev }
{@queue: queue1, @metric_type: max_memory, @value: 102400, @cluster: dev}
{@queue: queue2, @metric_type: used_memory, @value: 2048, @cluster: prod}
{@queue: queue2, @metric_type: max_memory, @value: 204800, @cluster: prod}

Result: (in a new index)
{@queue: queue1, @used_memory: 1024, @cluster: dev}
{@queue: queue1, @max_memory: 102400, @cluster: dev}
{@queue: queue2, @used_memory: 2048, @cluster: prod}
{@queue: queue2, @max_memory: 204800, @cluster: prod}

Here I want to use distinct values of @metric_type to be my new fields and value for it is the corresponding value from the @value field of source index.

I have tried the transform API but I don't think it still supports to create new terms. Or by any other means, it is not possible to achieve this result.

Please let me know if there is a way I can use the transform API for this or any alternate solution.

Thanks.

[spinscale (Alexander Reelsen)]

Hey,

can you show in more detail what you tried, and what errors you are getting? Which requests are you sending, what the responses are looking like and what you would expect them to be instead.

Otherwise debugging gets crazy hard with only a few bits of information available.

--Alex