In Logstash output elastisearch plugin is there a way to get response code from elastic after insert?

[Ullas7 (Ullas A K)]

I trying to insert a document in elasticDB using the following logstash output config

elasticsearch{
			hosts => "localhost:9200"
			index => "indexname"
			document_id => "%{conversationId}"
			# action => "create"
		 
		}

I will get a duplicate message from the source. I need to know whether last save was insert or update.

depending on the response from elastic I would like to store duplicate msg another output source.

I tried with action => "create" hoping to catch error response .

I want to handle duplicate message

 if not duplicate
    insert to elastic
   push to kafka

if duplicate
  update to elastic   # this is taken care by document_id
  don't post to kafka

Can someone assist me in this?

[Badger]

The elasticsearch output does not provide any status that you can test.

How about using a second pipeline that reads new messages from elasticsearch and writes them to kafka?

[Ullas7 (Ullas A K)]

@Badger thanks for the response. This is exactly what I did. Here I am facing another issue, I put scheduler to pick from EL every 1 minute and write it to Kafka, now the problem is I need to manually update each document as processed(some flag) so that it won't be picked next batch. I need to add a new flag every time I push to Kafka ( two output writes)
I have another problem If I update the new document(duplicate) in the first pipeline the flag will be overridden and it will be again picked by the second pipeline.
This can be avoided by using action => create