Logstash closes after reading ES index

(Vilas Reddy Podduturi) #1


I am trying to read an ES index. But I observe that Logstash closes immediately after reading the ES index. But I would like Logstash to wait for new records into ES. Is it the common behavior with ES as input to Logstash?

Please clarify.


(Ed) #2

Well Elasticsearch does not "wait" for new documents, it has to query (Scrolls help it not repeat the entire search)

You may want to look at watcher which is a new product by ES that might help you out.

(Mark Walkom) #3

This is also relevant - https://github.com/elastic/elasticsearch/issues/1242

(Ed) #4

Interesting discussion. Yes event oriented notifications would be nice! If I get the gist of the thread.

I guess it depends on how many connections your dealing with. I mean if we are worrying about 10 to 100 connections per minute it really is not a strain on client or server. but if your talking about a million or more then that is another story.

(Vilas Reddy Podduturi) #5

I was wondering why Elasticsearch doesn't wait for new documents? Can we do something to achieve that using Logstash?

I have already tried using watcher few days back. It is a very good product. We are considering the same. But we currently do posses in house alerting system which we would like to leverage on before we take such a decision to go with watcher.
Is there a possibility that I can wait on ES using Logstash?

I am still trying to follow the discussion.

(system) #6