APM custom pipeline with enrich processor does not work

[JasonREC]

Hi all, I am trying to use enrich processor in the injest pipeline and apply into apm custom pipeline, but it does not work.

Here is my set up,
source index:

Enrich process policy:

Injest pipeline name: limit_lookup with enrich processor:

And, I have applied this to my to apm custom pipeline

PUT _ingest/pipeline/traces-apm@custom
{
  "processors": [
    {
      "pipeline": {
        "name": "limit_lookup" 
      }
    }
  ]
}

Basically, as long as the incoming document that contains the labels_data_name is matched the labels_database_name that in the source index. I would like it to add that couple fields into the document before it write into index.

Here is my apm index data looks like, the incoming data have a field called labels_database_name like below:

[leandrojmp (Leandro Pereira)]

Something is not adding up, you are referring to a field name labels_database_name, but the screenshot you shared from your kibana does not have this field, it has a field named labels.database_name, which is completely different.

Can you go into Discover on Kibana, expand a document and share the Json of how this document looks like?

[leandrojmp (Leandro Pereira)]

Oh I see.

In your source index your field is labels_database_name, but from what you shared from Kibana the field in your document is labels.database_name, which is different, you need to use this in your enrich processor.

Try to change the the field in your enrich processor to labels.database_name.

[warkolm (Mark Walkom)]

Is this the same as Configuration of transform data with custom pipeline - #3 by JasonREC?

[JasonREC]

@leandrojmp
Oh! I didn't realize that when I upload my csv data to elastic, it automactially change my field from labels.database_name -> labels_database_name . I look at this issue in other discussion it is due to the elastic dot conflict problem.

so, I try to use update query to change the field name, but it failed due to Elastic consider the dot is implying I am accessing an object attribute instead of considering it is a part of name.

and then I try to add a rename processor before the enrich processor, so that I change the label.database_name back to label_database_name. but seems does not work also.

[JasonREC]

Hi, @leandrojmp

Never mind, I just found out there is a over write setting which can help me change the labels_database_name back to labels.database_name

Now, the enrich processor works as expected
Thank you