In the meanwhile, I spotted this post which allows me to use the Elasticsearch output plugin in upsert mode - which pretty near does what i need. Here's what my output section now looks like
...
output {
if "source1" in [tags] {
elasticsearch { ..} # write to source1 specific index
}
else if "source2" in [tags] {
elasticsearch { ..} # write to source2 specific index
}
# and ultimately the combined index
elasticsearch{
hosts => ["my-es-host:9200"]
index => ["my-combined-index"]
action => "update"
document_id => "%{Operation}_%{RefData1}_%{RefData1}"
doc_as_upsert => true
}
}
This gives me two choices - wicked!