How To read in logstash the response from elasticsearch using elasticsearch output plugin

[yeppazu]

Hi
there is a some method to read response from Elasticsearch and trigger an action over it? (eg: running script, call an http url, ...)

Normally, elastic return a JSON after HTTP index request. This JSON cotain some information, including "_id".

My need is to read the "_id" that elastic return after a "create" in elastic, and send it to another "software agent".
More in general, my interests is to read input message, hash it and save both (message and hash) into a my DB, with relative elastic _ID.

Something like this (it's pseudo code)

output {
    elasticsearch {
        hosts => "192.168.0.111"
        exec_after_return {
          command => "/opt/myscript.py %{message} %{[@response][_id]}"
        }
    }
}

Thanks & best regards,
Y.

[aaron-nimocks]

I don't believe the ES Output plugin can return the ID it just created.

Option 1 - Generate your own document ID with the UUID filter plugin or however you want to generate it and use it in your Elasticsearch output so you have it and can send to another output.

Option 2 - Create another Logstash pipeline that reads the Elasticsearch as an input and then you can get and use the document ID. Maybe add another field and write back to that Elasticsearch index to indicate it's been processed so the input query won't keep getting the same records.

[yeppazu]

Thanks @aaron-nimocks

can I ask to you a link for study in deep about "Logstash pipeline" to do what you said? (a input pipeline that is able to read "Elasticsearch output") ... this is very interesting for me

Thanks and best regards,
Y

[aaron-nimocks]

I don't think it would work quite how you are thinking.

I was more thinking you do a completely separate Logstash pipeline that has an Elasticsearch Input along with a query that will filter out any records you already processed.

input {
 elasticsearch {
  hosts => "localhost"
  query => '{"query":{"bool":{"must_not":{"exists":{"field":"processed"}}}}}'
  docinfo => true
 }
}

Now you have access to a field %{[@metadata][_id]} that you can use the ID for and can send to your other agent in the output.

But I would also have another output to Elasticsearch that will add a field called processed so the input doesn't process the document again.

output {
 elasticsearch {
  hosts => "localhost"
  index => "%{[@metadata][_index]}"
  document_id => "%{[@metadata][_id]}"
  add_field => { "processed" => "true" }
 }
}

[yeppazu]

Thanks a lot

so, after "Elasticsearch output" I can use a "exec output" plugin with "message" and the "id" retived from elastic.

Only dubt is that in this pipeline I dont have my original "input" (some logs come from somewhere), so I don't have the "message".

if I'm not wrong, I can add another input plugin following Elasticsearch?

I suppose that the more efficient and easy strategy is your first Option

[aaron-nimocks]

I don't know if what you are suggesting is possible.

I would recommend my first option since it's easiest but will only work with newly ingested records. The second option is more robust and will work with all records currently in the index and any new processed ones.