Multiple output to same elasticsearch in order, one for delete action, another for normal action(create if)

See detail in https://stackoverflow.com/questions/49441683/logstash-elasticsearch-how-can-i-remove-all-data-before-output-new-data

I just want to delete old data before output new data.

So I am curious, if I define two outputs to same elasticsearch, one for delete action, another for normal action, what will happen? Can logstash ensure to run output in order?

output {
  elasticsearch {
    hosts => ["docker.for.mac.localhost:19200"]
    index => "osquery-%{host}"
    action => "delete"
  }
  elasticsearch {
    hosts => ["docker.for.mac.localhost:19200"]
    index => "osquery-%{host}"
  }
}

EDIT@2018-03-28

I'v solved this issue https://stackoverflow.com/a/49504227/2293666, so no need to talk about this topic.

Logstash seems run output in parallel, not possible to output one by one in order defined in the conf.

Why can't you just update the existing document? I don't see the point in deleting and recreating it.

If you want to have one document for each host+package, then set the document_id to be a function (possibly the concatetation of) host+package. Then it will keep overwriting the documents each time you get a new set of data.

Actually at first I tried to overwrite existing document. But soon I found the problem is that the output is multiple documents, not a single document, sometimes more, sometime less, so it is not a simple update. It must be a remove all & add.

And the object inside array is not a fixed schema,
I can not assume which field is key,
and to be able to search inside array precisely,
I need to split array into multiple documents(events) instead of specify snapshot as "nested" object in elasticsearch mapping.

Someday:
{packages: [{name:"pkg1", version:"1"}, {name:"pkg2", version:"2"}]}
{...: [{...}, {...}]}

Another day:
{mounts: [{path:"path1", device:"dev1"}, {path:"path2", device:"dev2"]}
{...: [{...}, {...}]}

Another day:
{modules: [{name:"mod1", desc:"desc1"}, {name:"mod1", desc:"desc1"]}
{...: [{...}, {...}]}

Another day:
{packages: [{name:"pkg1", version:"1"}, {name:"pkg2", version:"2"}, {name:"pkg3", version:"3"}]}
{...: [{...}, {...}]}

Thank you. Actually at first I tried to overwrite existing document. But soon I found the problem is that the output is multiple documents, not a single document, sometimes more, sometime less, so it is not a simple update. It must be a remove all & add. And the object inside array is not a fixed schema,
I can not assume which field is key.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.