Elasticsearch percolator and Logstash ... How to?


(Thorsten Nickel) #1

Dear all,

this is my first post to this group, since any more googling does not
result in any more results, this looks like my last hope.

Regarding my problem, I have an ELK Stack up and running. My logstasher
sits on system logs, grok's them as I want them to, and puts these
informations
via the elasticsearch output plugin into ES. Using Kibana Frontend, I can
see my Loglines flowing in, and everything is good.

Now for my scenario, I would like to have some form of basic 'alerting' put
into this chain. For this, I put up a percolator query into ES.
If I use CURL to ask with this percolator, I get my result like this :

curl -XGET
'127.0.0.1:9200/logstash-2014.12.05/MONITOR/AUoaylhicoeQR6lWzrbU/_percolate?pretty'
{
"took" : 38,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"total" : 1,
"matches" : [ {
"_index" : "logstash-2014.12.05",
"_id" : "monitor_red"
} ]
}

The point is, I kinda need to do this percolator query, when I put my Doc
into ES using logstash so I can see, if it triggers the percolator query or
not. Or another point would be, if I have like
thousands of docs in ES, how would I percolate through all of them to see,
which ones match the monitor_red query?

As far as I can see, I need to query with the percolator one doc at a
time.... looks really cumbersome to me.

I hope that I explained my problem good enough.

Kind regards,
T. Nickel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8a422d3a-2d8a-4bb3-9e2d-a78dff154c2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jay Swan) #2

Logstash doesn't support the percolator API today.

On Friday, December 5, 2014 8:02:39 AM UTC-7, Thorsten Nickel wrote:

Dear all,

this is my first post to this group, since any more googling does not
result in any more results, this looks like my last hope.

Regarding my problem, I have an ELK Stack up and running. My logstasher
sits on system logs, grok's them as I want them to, and puts these
informations
via the elasticsearch output plugin into ES. Using Kibana Frontend, I can
see my Loglines flowing in, and everything is good.

Now for my scenario, I would like to have some form of basic 'alerting'
put into this chain. For this, I put up a percolator query into ES.
If I use CURL to ask with this percolator, I get my result like this :

curl -XGET '
127.0.0.1:9200/logstash-2014.12.05/MONITOR/AUoaylhicoeQR6lWzrbU/_percolate?pretty
'
{
"took" : 38,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"total" : 1,
"matches" : [ {
"_index" : "logstash-2014.12.05",
"_id" : "monitor_red"
} ]
}

The point is, I kinda need to do this percolator query, when I put my Doc
into ES using logstash so I can see, if it triggers the percolator query or
not. Or another point would be, if I have like
thousands of docs in ES, how would I percolate through all of them to see,
which ones match the monitor_red query?

As far as I can see, I need to query with the percolator one doc at a
time.... looks really cumbersome to me.

I hope that I explained my problem good enough.

Kind regards,
T. Nickel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5de53375-4078-4f98-bbfc-4b3fe2c0f485%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Val Crettaz) #3

There was a similar need reported here and I answered there.

Such a Logstash filter now exists, it has not been released on RubyGems but you can easily build it and install it.


#4

I have exactly the same problem. The perculator filter is stil not included in logstash... Is there a way to perculate with logstash <> elasticsearch in 5.x?


(system) #5