ELSER ingest pipeline with ingest processor

Elastic Stack Elasticsearch
1

I have peculiar situation where I need to create embeddings for each text value of a field within an array of objects.

Sample Document:

{
"areas_of_expertise":[{
              "id": 14,
              "is_active": true,
              "is_delete": false,
              "created_on": "2023-12-21T06:56:00.040034+00:00",
              "updated_on": "2023-12-21T06:56:00.040034+00:00",
              "created_by": {},
              "updated_by": {},
              "deleted_by": {},
              "uuid": "f6ddf7d4-69d0-4385-a06e-0edd273b0241",
              "area_of_expertise": "Agricultural and natural resource economics",
              "is_primary": false,
              "reviewed_by_expert": false,
              "expert_comment": null,
              "researcher_comment": null,
              "source": "expert cv",
              "display_setting": "Private"
            },
            {
              "id": 15,
              "is_active": true,
              "is_delete": false,
              "created_on": "2023-12-21T06:56:00.040034+00:00",
              "updated_on": "2023-12-21T06:56:00.040034+00:00",
              "created_by": {},
              "updated_by": {},
              "deleted_by": {},
              "uuid": "6ba60787-e259-4963-be53-5166820f5f68",
              "area_of_expertise": "econometrics",
              "is_primary": false,
              "reviewed_by_expert": false,
              "expert_comment": null,
              "researcher_comment": null,
              "source": "expert cv",
              "display_setting": "Private"
            },
            {
              "id": 16,
              "is_active": true,
              "is_delete": false,
              "created_on": "2023-12-21T06:56:00.040034+00:00",
              "updated_on": "2023-12-21T06:56:00.040034+00:00",
              "created_by": {},
              "updated_by": {},
              "deleted_by": {},
              "uuid": "6d9efa04-aff8-4394-90f0-38737dfdd90e",
              "area_of_expertise": "welfare economics",
              "is_primary": false,
              "reviewed_by_expert": false,
              "expert_comment": null,
              "researcher_comment": null,
              "source": "expert cv",
              "display_setting": "Private"
            }]
}

Based on the above document, I need embedding for each of areas_of_expertise.area_of_expertise

The ingest pipeline I am using is as below:

curl -XPUT "http://<elastic-host>:9200/_ingest/pipeline/elser-ingest-pipeline?pretty" -H "kbn-xsrf: reporting" -H "Content-Type: application/json" -d'
{
  "description": "Pipeline to embed area of expertise",
  "processors": [
    {
      "foreach": {
        "field": "areas_of_expertise",
        "processor": {
          "inference": {
            "model_id": ".elser_model_2",
            "input_output": [
              {
                "input_field": "_ingest._value.area_of_expertise",
                "output_field": "_ingest._value.area_of_expertise_embedding"
              }
            ]
          }
        }
      }
    }
  ]
}'

Doesn't seem to work. Any help is appreciated. Cheers!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.