Index Mappings over nested fields

[Fabio-sama (Fabio)]

Hi there!

I'd like to know whether it is possible to apply an index mapping so to manipulate some nested fields as soon as they arrive.

To make it more understandable, let's say I have some in-coming documents structured as follows:

field1: ...
field2: ...
field3: 
[
    {
        nestedfield1: ...
        nestedfield2: float
        nestedfield3: ...
    },
    {
        nestedfield1: ...
        nestedfield2: float
        nestedfield3: ...
    },
    .
    .
    .
    {
        nestedfield1: ...
        nestedfield2: float
        nestedfield3: ...
    }
]

I'd like to add to such documents a further NON nested field (i.e. sibling of field1, field2, field3) which is the sum of the nestedfield2s (e.g. total_nestedfield2: float).

Docs are ingested from a third party platform so I cannot modify the way they are generated but only manipulate them at load time.

Ideas?
Thank you!

[Christian_Dahlqvist]

Mappings do not alter the document in any way. If you want to change the structure you will need to use an ingest pipeline that you link to the index.

[Fabio-sama (Fabio)]

First of all thank you for the fast reply.

Anyway, I've used ingest pipelines with _update_by_query in the past to modify some historical data but as far as I know they cannot access nested fields (in fact I had to use Logstash with a ruby script to fulfill the task eventually).

Do you know any other way to achieve such a goal?

[Christian_Dahlqvist]

I suspect you would need to use a script process or with Painless in the pipeline. Otherwise you may need to find a way to get the data through Logstash, e.g. by using a http input and bulk codec.

[Fabio-sama (Fabio)]

Mhmh don't know if I can redirect the output of the third party app to logstash, I'll check that out.

As for now, I'll try with a painless script even though I'm not sure it'll let me access nested fields and make computation over them.

I'll let you know (and ask for help) asap!

Thank you for now