Elasticsearch Filter: Facing issue when using elasticsearch filter with logstash

[rainman]

I am using Logstash to pull data from the Oracle database and process some validation and index into Elasticsearch.

I have to update the existing document in the index if the "ID" exists. For that, I am using Elasticsearch Filter to check if the id exists and pull only one doc if the id exists.

Everything is working as expected. But because of bulk request we are missing the updates for some events,

1. Does Logstash has different threads to process every event? or
2. How to handle the events with elastisearch filter plugin?

[BenB196]

I'm not sure if below is the best option, but it could work.

If you set your ID field to _id. You can then use the Logstash Elasticsearch output, with action set to update. When indexing elasticsearch will check the _id field to see if it exists, if it does, it will update, if it doesn't it will create the doc.

Like I said I'm not 100% sure this is the best option, someone else might have a better solution.

[rainman]

Please see my below configs. this will be helpful to suggest to me.

COSFitler is my internal plugin.

filter {
	elasticsearch {
		hosts => ["localhost"]
		index => "autoupdate"
		query_template => "/etc/logstash/query/query-template.json"
		fields => {
			"id" => "idupdate"
			"programname" => "programnameold"
			"soccode" => "soccodeold"
			"soctitle" => "soctitleold"
			"onetcode" => "onetcodeold"
			"onettitle" => "onettitleold"
		}
	}
	cosfilter {
		combine => { "programname" => [ "programname","programnameold" ] }
	}
}

output {

	if[idupdate] {
		elasticsearch {
			hosts => "localhost"
			index => "autoupdate"
			document_id => "%{id}"
			manage_template => true
			action => "update"
    	}
	}

    elasticsearch {
		hosts => "localhost"
		index => "autoupdate"
		document_id => "%{id}"
		manage_template => true
		action => "index"
    }
}

[rainman]

query-template.json

{
  "size": 1,
  "query": { "match":{"id": "%{[id]}" } }
}

[rainman]

@Badger/ @magnusbaeck Can you please help with this sceario.

[BenB196]

I think you're over complicating your output. You can just have a singular output with action update. This will work via upsert to index any new documents.

[rainman]

I'm Sure, I have tried. I will give other try.

[rainman]

I feel the main problem is event is posted before the completion of the Elasticsearch filter.

[ritchierich (Rich Horace)]

If you use doc_as_upsert in the logstash elasticsearch output the will update the existing document or create a new one.

To test removed the elasticsearch filter and change your output to:

    elasticsearch {
        hosts => "localhost"
        index => "autoupdate"
        document_id => "%{id}"
        manage_template => true
        doc_as_upsert => "true"
        action => "update"
    }

[rainman]

Thank you,

But we have to keep the old data with the new data. I mean for a couple of fields we need to add the new data with the existing data.

for example Onettitle is array object second time the new string will get added. Thats how our data is.

FYI: We tried to process from db but it is not working. so I have started to look this way.