# Elasticsearch Filter: Facing issue when using elasticsearch filter with logstash

**URL:** <https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227>\
**Category:** Elasticsearch\
**Created:** [November 12, 2020, 2:42pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227 "2020-11-12T14:42:43Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [November 12, 2020, 2:42pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/1 "2020-11-12T14:42:43Z")

</div>

I am using Logstash to pull data from the Oracle database and process some validation and index into Elasticsearch.

I have to update the existing document in the index if the "ID" exists. For that, I am using Elasticsearch Filter to check if the id exists and pull only one doc if the id exists.

Everything is working as expected. But because of bulk request we are missing the updates for some events,

1. Does Logstash has different threads to process every event? or
2. How to handle the events with elastisearch filter plugin?

---

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [November 12, 2020, 2:55pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/2 "2020-11-12T14:55:27Z")

</div>

I'm not sure if below is the best option, but it could work.

If you set your ID field to \_id. You can then use the Logstash Elasticsearch output, with action set to update. When indexing elasticsearch will check the \_id field to see if it exists, if it does, it will update, if it doesn't it will create the doc.

Like I said I'm not 100% sure this is the best option, someone else might have a better solution.

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [November 12, 2020, 4:15pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/3 "2020-11-12T16:15:41Z")

</div>

Please see my below configs. this will be helpful to suggest to me.

COSFitler is my internal plugin.

```auto
filter {
	elasticsearch {
		hosts => ["localhost"]
		index => "autoupdate"
		query_template => "/etc/logstash/query/query-template.json"
		fields => {
			"id" => "idupdate"
			"programname" => "programnameold"
			"soccode" => "soccodeold"
			"soctitle" => "soctitleold"
			"onetcode" => "onetcodeold"
			"onettitle" => "onettitleold"
		}
	}
	cosfilter {
		combine => { "programname" => ["programname","programnameold"] }
	}
}

output {

	if[idupdate] {
		elasticsearch {
			hosts => "localhost"
			index => "autoupdate"
			document_id => "%{id}"
			manage_template => true
			action => "update"
    	}
	}

    elasticsearch {
		hosts => "localhost"
		index => "autoupdate"
		document_id => "%{id}"
		manage_template => true
		action => "index"
    }
}

```

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [November 12, 2020, 4:17pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/4 "2020-11-12T16:17:19Z")

</div>

query-template.json

```auto
{
  "size": 1,
  "query": { "match":{"id": "%{[id]}" } }
}

```

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [November 12, 2020, 4:19pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/5 "2020-11-12T16:19:23Z")

</div>

@Badger/ @magnusbaeck Can you please help with this sceario.

---

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [November 12, 2020, 4:23pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/6 "2020-11-12T16:23:40Z")

</div>

I think you're over complicating your output. You can just have a singular output with action update. This will work via upsert to index any new documents.

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [November 12, 2020, 4:27pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/7 "2020-11-12T16:27:56Z")

</div>

I'm Sure, I have tried. I will give other try.

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [November 12, 2020, 4:39pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/8 "2020-11-12T16:39:34Z")

</div>

I feel the main problem is event is posted before the completion of the Elasticsearch filter.

---

<div class="post-metadata">

**Author:** ![ritchierich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ritchierich/32/4329_2.png) [@ritchierich](https://discuss.elastic.co/u/ritchierich)\
**Post date:** [November 12, 2020, 5:06pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/9 "2020-11-12T17:06:45Z")

</div>

If you use [doc\_as\_upsert](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-doc_as_upsert) in the logstash elasticsearch output the will update the existing document or create a new one.

To test removed the elasticsearch filter and change your output to:

```auto
    elasticsearch {
        hosts => "localhost"
        index => "autoupdate"
        document_id => "%{id}"
        manage_template => true
        doc_as_upsert => "true"
        action => "update"
    }

```

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [November 12, 2020, 5:15pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/10 "2020-11-12T17:15:43Z")

</div>

Thank you,

But we have to keep the old data with the new data. I mean for a couple of fields we need to add the new data with the existing data.

for example **Onettitle** is array object second time the new string will get added. Thats how our data is.

FYI: We tried to process from db but it is not working. so I have started to look this way.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 10, 2020, 5:16pm UTC](https://discuss.elastic.co/t/elasticsearch-filter-facing-issue-when-using-elasticsearch-filter-with-logstash/255227/11 "2020-12-10T17:16:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.