I am trying to read an ES index. But I observe that Logstash closes immediately after reading the ES index. But I would like Logstash to wait for new records into ES. Is it the common behavior with ES as input to Logstash?
Well Elasticsearch does not "wait" for new documents, it has to query (Scrolls help it not repeat the entire search)
You may want to look at watcher which is a new product by ES that might help you out.
Interesting discussion. Yes event oriented notifications would be nice! If I get the gist of the thread.
I guess it depends on how many connections your dealing with. I mean if we are worrying about 10 to 100 connections per minute it really is not a strain on client or server. but if your talking about a million or more then that is another story.
I was wondering why Elasticsearch doesn't wait for new documents? Can we do something to achieve that using Logstash?
I have already tried using watcher few days back. It is a very good product. We are considering the same. But we currently do posses in house alerting system which we would like to leverage on before we take such a decision to go with watcher.
Is there a possibility that I can wait on ES using Logstash?
I am still trying to follow the discussion.