I noticed that occasionally I need to shield my ES cluster from some
documents, which are too many or too big or otherwise poison ES.
Usually I can formulate pretty easy query or criteria to detect those
documents and I'm looking for a way to block them from entering the index.
Is there such pre-indexing filtering mechanism? May be Transforms can be
used for that purpose?
Best way to do it is on the client side I believe. You could probably
abuse transforms to just blow up when you see something you don't like. I
don't think they have the ability to manipulate the operation (to make it
noop) though. If they do there certainly aren't any tests to make sure
that that doesn't break.
Nik
On Fri, Dec 12, 2014 at 5:11 PM, Konstantin Erman konste@gmail.com wrote:
I noticed that occasionally I need to shield my ES cluster from some
documents, which are too many or too big or otherwise poison ES.
Usually I can formulate pretty easy query or criteria to detect those
documents and I'm looking for a way to block them from entering the index.
Is there such pre-indexing filtering mechanism? May be Transforms can be
used for that purpose?
Why are you putting business logic of this type in ES? It belongs in your
workflow. At the ES indexer level you will have no idea of the source of
truth of the questionable content. Unless you're web crawliing which means
you're using the wrong search platform altogether imo.
On Friday, December 12, 2014 5:11:05 PM UTC-5, Konstantin Erman wrote:
I noticed that occasionally I need to shield my ES cluster from some
documents, which are too many or too big or otherwise poison ES.
Usually I can formulate pretty easy query or criteria to detect those
documents and I'm looking for a way to block them from entering the index.
Is there such pre-indexing filtering mechanism? May be Transforms can be
used for that purpose?
I don't crawl the web, just collect rather verbose logs from multiple
private cloud services and try to keep the size of ES cluster just
sufficient for comfortable searching those logs. Monitored services are
under development and occasionally (because of bugs or specifics of the
source data) they start to send orders of magnitude higher than usual
torrent of logs. When this happens, very soon ES cluster become
non-responsive and drops logs from all services, bad behaving or not.
We cannot afford to keep the cluster of the size capable to handle those
peak loads (and idling most of the time). We rather need some kind of
Denial of Service attack prevention logic. When some client(s) goes over
its quota of logs it should be blocked, rather than melting cluster down.
River plugin looks like overkill to me, especially considering deprecation
of rivers.
On Saturday, December 13, 2014 7:33:05 PM UTC-8, BillyEm wrote:
Why are you putting business logic of this type in ES? It belongs in your
workflow. At the ES indexer level you will have no idea of the source of
truth of the questionable content. Unless you're web crawliing which means
you're using the wrong search platform altogether imo.
On Friday, December 12, 2014 5:11:05 PM UTC-5, Konstantin Erman wrote:
I noticed that occasionally I need to shield my ES cluster from some
documents, which are too many or too big or otherwise poison ES.
Usually I can formulate pretty easy query or criteria to detect those
documents and I'm looking for a way to block them from entering the index.
Is there such pre-indexing filtering mechanism? May be Transforms can be
used for that purpose?
We solve problems like this in two ways:
Adding queueing or concurrent request limits.
Queueing buys retries for free and can absorb temporary shocks. You can
also get things like priority, backlog monitoring, and manual backlog
grooming. I think logstash already supports this, but I don't know it very
well.
Concurrent request limits are more brutal. You just throw away requests to
index if there are too many in flight. You can make it more granular by
giving each incoming application its own pool and limits. We implement
these using a simple server called poolcounter. You can find it by
searching for WMF poolcounterd.
Either way you would have to implement a small application to get these
integrated. Well, maybe someone has already made the queueing one, I don't
know.
Nik
On Dec 13, 2014 11:21 PM, "Konstantin Erman" konste@gmail.com wrote:
I don't crawl the web, just collect rather verbose logs from multiple
private cloud services and try to keep the size of ES cluster just
sufficient for comfortable searching those logs. Monitored services are
under development and occasionally (because of bugs or specifics of the
source data) they start to send orders of magnitude higher than usual
torrent of logs. When this happens, very soon ES cluster become
non-responsive and drops logs from all services, bad behaving or not.
We cannot afford to keep the cluster of the size capable to handle those
peak loads (and idling most of the time). We rather need some kind of
Denial of Service attack prevention logic. When some client(s) goes over
its quota of logs it should be blocked, rather than melting cluster down.
River plugin looks like overkill to me, especially considering deprecation
of rivers.
On Saturday, December 13, 2014 7:33:05 PM UTC-8, BillyEm wrote:
Why are you putting business logic of this type in ES? It belongs in your
workflow. At the ES indexer level you will have no idea of the source of
truth of the questionable content. Unless you're web crawliing which means
you're using the wrong search platform altogether imo.
On Friday, December 12, 2014 5:11:05 PM UTC-5, Konstantin Erman wrote:
I noticed that occasionally I need to shield my ES cluster from some
documents, which are too many or too big or otherwise poison ES.
Usually I can formulate pretty easy query or criteria to detect those
documents and I'm looking for a way to block them from entering the index.
Is there such pre-indexing filtering mechanism? May be Transforms can be
used for that purpose?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.