# Set \_id during pipeline in bulk ingestion

**URL:** <https://discuss.elastic.co/t/set-id-during-pipeline-in-bulk-ingestion/357158>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [April 10, 2024, 2:53pm UTC](https://discuss.elastic.co/t/set-id-during-pipeline-in-bulk-ingestion/357158 "2024-04-10T14:53:48Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![wnmills3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wnmills3/32/133048_2.png) [@wnmills3](https://discuss.elastic.co/u/wnmills3)\
**Post date:** [April 10, 2024, 2:53pm UTC](https://discuss.elastic.co/t/set-id-during-pipeline-in-bulk-ingestion/357158/1 "2024-04-10T14:53:48Z")

</div>

How can I set the \_id field to the value of the id property defined in my mappings?

In my mappings I have an id field defined as follows:

```auto
			"id": {
				"type": "text",
				"fields": {
					"type": "keyword",
					"ignore_above": 512
				},
				"store": "true"
			},

```

and I want the \_id to be set to its value. I've added a set processor as follows at the end of the array of processors:

```auto
    {
      "set": {
        "field": "_id",
        "copy_from": "id"
      }
    }

```

Note: I've also tried value rather than copy\_from...

However, I am getting errors stating:  
Input field [text\_field] does not exist in the source document

I've tried using different ways to reference the value I want the \_id to receive:  
{{id}} or {{{id}}} or "id"

My mapping for the \_source simply has:

```auto
		"_source": {
			"enabled": "true",
			"excludes": [
				"passage_embedding.predicted_value",
				"path_embedding.predicted_value"
			]
		},

```

as the id field is defined as an object in the properties.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 10, 2024, 3:26pm UTC](https://discuss.elastic.co/t/set-id-during-pipeline-in-bulk-ingestion/357158/2 "2024-04-10T15:26:40Z")

</div>

Is the id field located at the root level of the document? Can you show us a sample document?

---

<div class="post-metadata">

**Author:** ![wnmills3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wnmills3/32/133048_2.png) [@wnmills3](https://discuss.elastic.co/u/wnmills3)\
**Post date:** [April 10, 2024, 3:38pm UTC](https://discuss.elastic.co/t/set-id-during-pipeline-in-bulk-ingestion/357158/3 "2024-04-10T15:38:52Z")

</div>

```auto
        ObjectNode content = s_objectMapper
            .createObjectNode();
        content.put("passage", text);
        content.put("path", path);
        content.put("url", url);
        content.put("title", title);
        content.put("linenum", linenum);
        content.put("filestem", filestem);
        content.put("id", id);
        content.put("language", langFamily);
        ingester.add(op -> op.index(
            idx -> idx.index(indexName).document(content)));

```

---

<div class="post-metadata">

**Author:** ![wnmills3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wnmills3/32/133048_2.png) [@wnmills3](https://discuss.elastic.co/u/wnmills3)\
**Post date:** [April 10, 2024, 3:40pm UTC](https://discuss.elastic.co/t/set-id-during-pipeline-in-bulk-ingestion/357158/4 "2024-04-10T15:40:23Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af73cfc251ba6d3cebd97e52110a3ed1bbd2352f.png)

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [April 11, 2024, 4:56am UTC](https://discuss.elastic.co/t/set-id-during-pipeline-in-bulk-ingestion/357158/5 "2024-04-11T04:56:33Z")

</div>

What you're doing should work (and works for me).

Can you post a complete working example so that we can diagnose where the issue might be.

This is what I tested (on 8.11.3)

```auto
PUT /index-1
{
  "mappings": { "properties": { "id": { "type": "keyword" }} }
}

```

```auto
PUT /_ingest/pipeline/test
{
  "processors": [
      { "set": { "field": "_id", "copy_from": "id" } }
  ]
}

```

```auto
PUT /index-1/_doc/1?pipeline=test
{
  "id" : "abc"
}

```

```auto
GET /index-1/_search

===
{
  "took": 181,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": "index-1",
        "_id": "abc",
        "_score": 1.0,
        "_source": {
          "id": "abc"
        }
      }
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![wnmills3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wnmills3/32/133048_2.png) [@wnmills3](https://discuss.elastic.co/u/wnmills3)\
**Post date:** [April 11, 2024, 12:04pm UTC](https://discuss.elastic.co/t/set-id-during-pipeline-in-bulk-ingestion/357158/6 "2024-04-11T12:04:10Z")

</div>

Indeed, changing to use type keyword worked. What I'd had before was:

```auto

"id": {

"type": "text",

"fields": {

"type": "keyword",

"ignore_above": 512

},

"store": "true"

},

```

but now just:

```auto

"id": {

"type": "keyword"

},

```