We recently started using Elasticsearch percolator and when we pushed code to production it is taking too long to match a document against registered queries in percolator index. Here is some information:
Percolator index: 10 shards, 2 replicas
4.5M registered queries in the index.
Percolator queries are setup on their own index and separate from data documents.
Do not know the exact number but good number of registered queries have nested fields.
Our goal is to be able to match 1K-2K documents every 30 minutes against the registered 4.5M queries. We have tried applying filtering that has helped to some extent but still not too helpful as matching trying to match a document against queries using filtering is taking 30+ seconds. Any recommendation to speed things up here?
We are using Elasticsearch 5.1.