# How To read in logstash the response from elasticsearch using elasticsearch output plugin

**URL:** <https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135>\
**Category:** Logstash\
**Created:** [November 1, 2021, 11:50am UTC](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135 "2021-11-01T11:50:47Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![yeppazu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yeppazu/32/96595_2.png) [@yeppazu](https://discuss.elastic.co/u/yeppazu)\
**Post date:** [November 1, 2021, 11:50am UTC](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135/1 "2021-11-01T11:50:48Z")

</div>

Hi 🙂  
there is a some method to read response from Elasticsearch and trigger an action over it? (eg: running script, call an http url, ...)

Normally, elastic return a JSON after HTTP index request. This JSON cotain some information, including "\_id".

My need is to read the "\_id" that elastic return after a "create" in elastic, and send it to another "software agent".  
More in general, my interests is to read input message, hash it and save both (message and hash) into a my DB, with relative elastic \_ID.

Something like this (it's pseudo code)

```auto
output {
    elasticsearch {
        hosts => "192.168.0.111"
        exec_after_return {
          command => "/opt/myscript.py %{message} %{[@response][_id]}"
        }
    }
}

```

Thanks & best regards,  
Y.

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [November 1, 2021, 1:28pm UTC](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135/2 "2021-11-01T13:28:16Z")

</div>

I don't believe the ES Output plugin can return the ID it just created.

Option 1 - Generate your own [document ID](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-document_id) with the [UUID filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-uuid.html) or however you want to generate it and use it in your Elasticsearch output so you have it and can send to another output.

Option 2 - Create another Logstash pipeline that reads the Elasticsearch as an input and then you can get and use the [document ID](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch-docinfo). Maybe add another field and write back to that Elasticsearch index to indicate it's been processed so the input query won't keep getting the same records.

---

<div class="post-metadata">

**Author:** ![yeppazu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yeppazu/32/96595_2.png) [@yeppazu](https://discuss.elastic.co/u/yeppazu)\
**Post date:** [November 1, 2021, 1:39pm UTC](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135/3 "2021-11-01T13:39:31Z")

</div>

Thanks @aaron-nimocks 🙂

can I ask to you a link for study in deep about "Logstash pipeline" to do what you said? (a input pipeline that is able to read "Elasticsearch output") ... this is very interesting for me 🙂

Thanks and best regards,  
Y

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [November 1, 2021, 1:49pm UTC](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135/4 "2021-11-01T13:49:54Z")

</div>

I don't think it would work quite how you are thinking.

I was more thinking you do a completely separate Logstash pipeline that has an Elasticsearch Input along with a query that will filter out any records you already processed.

```auto
input {
 elasticsearch {
  hosts => "localhost"
  query => '{"query":{"bool":{"must_not":{"exists":{"field":"processed"}}}}}'
  docinfo => true
 }
}

```

Now you have access to a field `%{[@metadata][_id]}` that you can use the ID for and can send to your other agent in the output.

But I would also have another output to Elasticsearch that will add a field called `processed` so the input doesn't process the document again.

```auto
output {
 elasticsearch {
  hosts => "localhost"
  index => "%{[@metadata][_index]}"
  document_id => "%{[@metadata][_id]}"
  add_field => { "processed" => "true" }
 }
}

```

---

<div class="post-metadata">

**Author:** ![yeppazu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yeppazu/32/96595_2.png) [@yeppazu](https://discuss.elastic.co/u/yeppazu)\
**Post date:** [November 1, 2021, 1:57pm UTC](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135/5 "2021-11-01T13:57:23Z")

</div>

Thanks a lot 🙂

so, after "Elasticsearch output" I can use a "exec output" plugin with "message" and the "id" retived from elastic.

Only dubt is that in this pipeline I dont have my original "input" (some logs come from somewhere), so I don't have the "message".

if I'm not wrong, I can add another input plugin following Elasticsearch?

I suppose that the more efficient and easy strategy is your first Option 🙂

---

<div class="post-metadata">

**Author:** ![aaron-nimocks](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron-nimocks/32/73965_2.png) [@aaron-nimocks](https://discuss.elastic.co/u/aaron-nimocks)\
**Post date:** [November 1, 2021, 2:16pm UTC](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135/6 "2021-11-01T14:16:31Z")

</div>

I don't know if what you are suggesting is possible.

I would recommend my first option since it's easiest but will only work with newly ingested records. The second option is more robust and will work with all records currently in the index and any new processed ones.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 29, 2021, 2:17pm UTC](https://discuss.elastic.co/t/how-to-read-in-logstash-the-response-from-elasticsearch-using-elasticsearch-output-plugin/288135/7 "2021-11-29T14:17:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.