# How to delete a document in Elasticsearch using Logstash

**URL:** <https://discuss.elastic.co/t/how-to-delete-a-document-in-elasticsearch-using-logstash/196326>\
**Category:** Logstash\
**Created:** [August 22, 2019, 12:58pm UTC](https://discuss.elastic.co/t/how-to-delete-a-document-in-elasticsearch-using-logstash/196326 "2019-08-22T12:58:10Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Airn5475](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/airn5475/32/13757_2.png) [@Airn5475](https://discuss.elastic.co/u/Airn5475)\
**Post date:** [August 22, 2019, 12:58pm UTC](https://discuss.elastic.co/t/how-to-delete-a-document-in-elasticsearch-using-logstash/196326/1 "2019-08-22T12:58:10Z")

</div>

My ETL Stack is **JSON File =\> Filebeat =\> Logstash =\> Elasticsearch**  
I am trying to specify an action in my JSON object that is either **update** or **delete**

Here's a partial of my current JSON

```auto
{ "key":123,"index_name":"companies"}

```

Here's my current Pipeline .conf that does **inserts or updates just fine**

```auto
input { 
    beats {
        id => "filebeat-input"
        port => 5044
        codec => "json"
        include_codec_tag => false
    }
} 
output { 
    elasticsearch { 
      id => "elasticsearch-output"
      hosts => ["localhost:9200"]
      document_id => "%{key}"
      index => "%{[@metadata][index_name]}"
      action => "update"
      doc_as_upsert => true
      manage_template => false
    }
}

```

As mentioned, I now want to **delete** documents using this same pattern.

Updated JSON with new `action` field

```auto
{ "key":123,"index_name":"companies","action":"update"}
{ "key":123,"index_name":"companies","action":"delete"}

```

Updated Pipeline .conf file

```auto
output { 
    elasticsearch { 
      id => "elasticsearch-output"
      hosts => ["localhost:9200"]
      document_id => "%{key}"
      index => "%{[@metadata][index_name]}"
      action => "%{action}"
      doc_as_upsert => true
      manage_template => false
    }
}

```

My Elasticsearch ETL stack takes JSON objects from files using Filebeat and passes them through Logstash to Elasticsearch. I want to pass an action property in my JSON object that designates either "update" or "delete as the action I want to perform against the document in the index. My current config has the action hardwired to "update" and will do upserts just fine.

**JSON Sample**

```auto
{ "key":123,"index_name":"companies","action":"update"}
{ "key":123,"index_name":"companies","action":"delete"}

```

**Current Pipeline.conf**

```auto
input { 
    beats {
        id => "filebeat-input"
        port => 5044
        codec => "json"
        include_codec_tag => false
    }
} 
output { 
    elasticsearch { 
      id => "elasticsearch-output"
      hosts => ["localhost:9200"]
      document_id => "%{key}"
      index => "%{[@metadata][index_name]}"
      action => "update"
      doc_as_upsert => true
      manage_template => false
    }
}

```

I tried to do this:

```auto
output { 
    elasticsearch { 
      id => "elasticsearch-output"
      hosts => ["localhost:9200"]
      document_id => "%{key}"
      index => "%{[@metadata][index_name]}"
      action => "%{action}"
      doc_as_upsert => true
      manage_template => false
    }
}

```

When I run that conf and the document doesn't exist, it is not doing an update as expected.  
It throws this error:

> [2019-08-21T15:21:28,879][WARN][logstash.outputs.elasticsearch]  
> Could not index event to Elasticsearch.  
> {:status=\>404, :action=\>["update", {:\_id=\>"123", :\_index=\>"companies", :\_type=\>"\_doc", :routing=\>nil, :retry\_on\_conflict=\>1}, #],  
> :response=\>{"update"=\>{"\_index"=\>"companies", "\_type"=\>"\_doc", "\_id"=\>"123", "status"=\>404,  
> "error"=\>{"type"=\>"document\_missing\_exception", "reason"=\>"[\_doc][123]: document missing", "index\_uuid"=\>"uU9oXFtZSXGodoh70YG3Ng", "shard"=\>"0", "index"=\>"companies"}}}}

**Why are my upserts no longer working?**

I know I can use an if statement inside the .conf file and have two different output paths based on the `action` field, but I don't understand why the solution above doesn't work.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 19, 2019, 12:58pm UTC](https://discuss.elastic.co/t/how-to-delete-a-document-in-elasticsearch-using-logstash/196326/2 "2019-09-19T12:58:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.