Can I copy values from different field to a single field in Elastic

Elastic Stack Elasticsearch
1

Hi,

I want to copy values from multiple fields into single field using split pipeline.

Below is the pipeline I have created, I am copying field values from title and abstract to suggest field.

[
  {
    "split": {
      "field": "title",
      "separator": "\\s+",
      "target_field": "suggest",
      "ignore_missing": true
    }
  },
  {
    "split": {
      "field": "abstract",
      "separator": "\\s+",
      "target_field": "suggest",
      "ignore_missing": true
    }
  }
]

The issue I am facing here is the field values are getting ovverriden. for Example.

POST /_ingest/pipeline/split_pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "title": "value1 value2",
        "abstract": "value3 value4"
      }
    }
  ]
}

Above query is returning below output

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "abstract": "value3 value4",
          "suggest": [
            "value3",
            "value4"
          ],
          "title": "value1 value2"
        },
        "_ingest": {
          "timestamp": "2024-12-02T19:07:43.091134491Z"
        }
      }
    }
  ]
}

However I want the below output.

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "abstract": "value3 value4",
          "suggest": [
           "value1", 
            "value2",
            "value3",
            "value4"
          ],
          "title": "value1 value2"
        },
        "_ingest": {
          "timestamp": "2024-12-02T19:07:43.091134491Z"
        }
      }
    }
  ]
}

Can somebody help me how can I achieve it through split_pipeline (I don't want to use copy_To functionality)??

2

I believe you could use split to produce two different fields, then use the append processor to combine them: Append processor | Elasticsearch Guide [8.16] | Elastic

And then drop the extra field

3

Thanks You !!,

Please let me know if I am doing something wrong here.
Below is what you are trying to say right ?

PUT _ingest/pipeline/split_pipeline
{
  "description": "Split values from multiple fields into one target field based on space",
  "processors": [
  {
    "split": {
      "field": "title",
      "separator": "\\s+",
      "target_field": "titlecopy"
    }
  },
  {
    "split": {
      "field": "abstract",
      "separator": "\\s+",
      "target_field": "abstractcopy"
    }
  },
  {
    "append": {
      "field": "suggest",
      "value": [
        "{{{titlecopy}}}",
        "{{{abstractcopy}}}"
      ],
      "allow_duplicates": false
    }
  },
  {
    "remove": {
      "field": [
        "titlecopy",
        "abstractcopy"
      ],
      "ignore_missing": true
    }
  }
]
}

Testing pipeline

POST /_ingest/pipeline/split_pipeline/_simulate
{
  "docs": [
    {
      "_source": {
        "title": "value1 value2",
        "abstract": "value3 value4"
      }
    }
  ]
}

Getting below as output.

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_version": "-3",
        "_id": "_id",
        "_source": {
          "abstract": "value3 value4",
          "suggest": [
            "{0=value1, 1=value2}",
            "{0=value3, 1=value4}"
          ],
          "title": "value1 value2"
        },
        "_ingest": {
          "timestamp": "2024-12-02T20:11:36.015890973Z"
        }
      }
    }
  ]
}

However my desired output should be

 "suggest": [
            "value1", "value2",
            "value3", "value4"
          ],
4

Hmmm yeah, I misread the description though looking at this again you may be able to combine the foreach processor with the append processor to loop through the elements in field2 and append them to field1 (or a third field)

5

It worked!!!

Thank you very much

1 Like