Logstash closes after reading ES index

Elastic Stack Logstash
1

Hi,

I am trying to read an ES index. But I observe that Logstash closes immediately after reading the ES index. But I would like Logstash to wait for new records into ES. Is it the common behavior with ES as input to Logstash?

Please clarify.

Thanks.

2

Well Elasticsearch does not "wait" for new documents, it has to query (Scrolls help it not repeat the entire search)

You may want to look at watcher which is a new product by ES that might help you out.

3

This is also relevant - https://github.com/elastic/elasticsearch/issues/1242

4

Interesting discussion. Yes event oriented notifications would be nice! If I get the gist of the thread.

I guess it depends on how many connections your dealing with. I mean if we are worrying about 10 to 100 connections per minute it really is not a strain on client or server. but if your talking about a million or more then that is another story.

5

I was wondering why Elasticsearch doesn't wait for new documents? Can we do something to achieve that using Logstash?

I have already tried using watcher few days back. It is a very good product. We are considering the same. But we currently do posses in house alerting system which we would like to leverage on before we take such a decision to go with watcher.
Is there a possibility that I can wait on ES using Logstash?

I am still trying to follow the discussion.