How to set auto-generated _id to json field(csv column) in logstash file

[Ashutosh2808]

I've csv columns as below in logstash conf file:

filter{
csv {
            columns => ["ResolutionId","IssueId"]
            separator => ","
            skip_header => "true"
              
       }
            
    mutate {
        remove_field => [ "path", "host"]
        copy => {
          "ResolutionId" => "_id"
      
          }
     }
}

I want to store the value of auto-generated id(_id) that I see in Kibana into "ResolutionId". I've tried by using copy in mutate plugin but it doesn't stoe the value. Can someone please help me to achieve this?

[AquaX (Andreas Helmer)]

The _id is generated when the data is ingested into Elasticsearch.
What you can do is define an ingest pipeline in Elasticsearch that copies the _id to ResolutionID.

Ingest pipelines | Elasticsearch Guide [7.x] | Elastic

Or you can try and update the mapping of your destination index and use the copy_to parameter to copy the _id to ResolutionId

copy_to | Elasticsearch Guide [7.13] | Elastic

I have not tested this but the query for the mapping may look something like this:

PUT my-index-000001
{
  "mappings": {
    "properties": {
      "_id": {
        "copy_to": "ResolutionID" 
      },
      "ResolutionID": {
        "type": "text"
      }
    }
  }
}

[stephenb (Stephen Brown)]

Just Driving By....

@Ashutosh2808 Welcome to the Community

Or You could use a Runtime Field... not as efficient... but super easy!

In the Index Pattern Create One... and it can be used anywhere... just keep in mind since it is runtime it is not actually persisted.

[ritchierich (Rich Horace)]

@Ashutosh2808 - Welcome to the community!

Another option would be to assigned the document id in the elasticsearch output. This might give you extra flexibility to control it before being ingested into elasticsearch.

output {
  elasticsearch {
    hosts    => [ 'eshost:9200' ]
    index => "logstash-resolution"
    document_id => "%{ResolutionId}"
    doc_as_upsert => "true"
    action => "update"
  }
}