Elasticsearch percolator and Logstash ... How to?

Dear all,

this is my first post to this group, since any more googling does not
result in any more results, this looks like my last hope.

Regarding my problem, I have an ELK Stack up and running. My logstasher
sits on system logs, grok's them as I want them to, and puts these
informations
via the elasticsearch output plugin into ES. Using Kibana Frontend, I can
see my Loglines flowing in, and everything is good.

Now for my scenario, I would like to have some form of basic 'alerting' put
into this chain. For this, I put up a percolator query into ES.
If I use CURL to ask with this percolator, I get my result like this :

curl -XGET
'127.0.0.1:9200/logstash-2014.12.05/MONITOR/AUoaylhicoeQR6lWzrbU/_percolate?pretty'
{
"took" : 38,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"total" : 1,
"matches" : [ {
"_index" : "logstash-2014.12.05",
"_id" : "monitor_red"
} ]
}

The point is, I kinda need to do this percolator query, when I put my Doc
into ES using logstash so I can see, if it triggers the percolator query or
not. Or another point would be, if I have like
thousands of docs in ES, how would I percolate through all of them to see,
which ones match the monitor_red query?

As far as I can see, I need to query with the percolator one doc at a
time.... looks really cumbersome to me.

I hope that I explained my problem good enough.

Kind regards,
T. Nickel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8a422d3a-2d8a-4bb3-9e2d-a78dff154c2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Logstash doesn't support the percolator API today.

On Friday, December 5, 2014 8:02:39 AM UTC-7, Thorsten Nickel wrote:

Dear all,

this is my first post to this group, since any more googling does not
result in any more results, this looks like my last hope.

Regarding my problem, I have an ELK Stack up and running. My logstasher
sits on system logs, grok's them as I want them to, and puts these
informations
via the elasticsearch output plugin into ES. Using Kibana Frontend, I can
see my Loglines flowing in, and everything is good.

Now for my scenario, I would like to have some form of basic 'alerting'
put into this chain. For this, I put up a percolator query into ES.
If I use CURL to ask with this percolator, I get my result like this :

curl -XGET '
127.0.0.1:9200/logstash-2014.12.05/MONITOR/AUoaylhicoeQR6lWzrbU/_percolate?pretty
'
{
"took" : 38,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"total" : 1,
"matches" : [ {
"_index" : "logstash-2014.12.05",
"_id" : "monitor_red"
} ]
}

The point is, I kinda need to do this percolator query, when I put my Doc
into ES using logstash so I can see, if it triggers the percolator query or
not. Or another point would be, if I have like
thousands of docs in ES, how would I percolate through all of them to see,
which ones match the monitor_red query?

As far as I can see, I need to query with the percolator one doc at a
time.... looks really cumbersome to me.

I hope that I explained my problem good enough.

Kind regards,
T. Nickel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5de53375-4078-4f98-bbfc-4b3fe2c0f485%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

There was a similar need reported here and I answered there.

Such a Logstash filter now exists, it has not been released on RubyGems but you can easily build it and install it.

I have exactly the same problem. The perculator filter is stil not included in logstash... Is there a way to perculate with logstash <> elasticsearch in 5.x?