Custom _id modification in elastic search

Elastic Stack Elasticsearch
1

I want to set “_id” value in Elasticsearch document as my custom document id,
using this from kibana?
POST example/_doc
{
"_id":123,
"firstName":"Sera",
"lastName":"Zuck"
}
for this what i need to do ? should i have to modify the source code or should i have to include something else in connector?

my elastic-connector-config file is this:

 {
 "name": "elastic-sink-connector",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkCon>
    "tasks.max": "1",
    "topics": "test_topic",
    "connection.url": "http://localhost:9200",
    "key.ignore": "false",
    "schema.ignore": "true",
    "type.name": "_doc",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  }
}

PS: i dont want the _id feild to be autogenerated, i want to pass it as request. I am working with confluent 7.6.0, elasticSearch 8.12.2 , kibana 8.12.2

2

Welcome!

It should be something like:

PUT example/_doc/123
{
  "firstName":"Sera",
  "lastName":"Zuck"
}
3

Hi @dadoonet
Thank you for your response.
I wanted to seek clarification regarding the use of /_doc/something. Our current connector seems to require a different format, and I want to make sure I'm not missing any crucial details.
Could you please provide additional information or guide me on the preferred format for our connector?

4

I don't know as this is not made by us.
I guess you always could use in the worse case an ingest pipeline to modify your documents on the fly.

5

You don't need to modify the source code or add anything extra to the connector. Just ensure your Kafka message key contains the custom _id you want for your Elasticsearch document. The connector configuration you provided is already set up to use the Kafka message key as the Elasticsearch document _id with "key.ignore": "false" . Make sure your Kafka messages are produced with the correct key that represents your custom _id .

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.