I just want to delete old data before output new data.
So I am curious, if I define two outputs to same elasticsearch, one for delete action, another for normal action, what will happen? Can logstash ensure to run output in order?
If you want to have one document for each host+package, then set the document_id to be a function (possibly the concatetation of) host+package. Then it will keep overwriting the documents each time you get a new set of data.
Actually at first I tried to overwrite existing document. But soon I found the problem is that the output is multiple documents, not a single document, sometimes more, sometime less, so it is not a simple update. It must be a remove all & add.
And the object inside array is not a fixed schema,
I can not assume which field is key,
and to be able to search inside array precisely,
I need to split array into multiple documents(events) instead of specify snapshot as "nested" object in elasticsearch mapping.
Thank you. Actually at first I tried to overwrite existing document. But soon I found the problem is that the output is multiple documents, not a single document, sometimes more, sometime less, so it is not a simple update. It must be a remove all & add. And the object inside array is not a fixed schema,
I can not assume which field is key.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.