I recently deployed a new logging cluster on Kibana / ES 6.2.4
My old cluster was on 2.4
Performance on the new cluster is terribly slow. Queries that the 2.4 cluster is returning in a matter of 3-5 seconds are taking > 30 seconds (which is the default timeout) on my 6.2 cluster.
I am wondering if there are any major gotchas that I haven't considered while upgrading from 2.4 to 6.
In the browser, I see the _msearch network request either 502 or get cancelled.
It is using http/1.1 and no SSL. But my understanding is that ES doesn't support http/2, so would having an http/2 proxy in front of it yield performance improvements?
This is on a new cluster, it's receiving fresh data from fluent / td-agent -- I never tried to do a data dump from ES 2.4 to 6.2
Any help would be appreciated, I understand this is a bit vague of a description, I'm happy to provide more information!
This is running in Ubuntu 16.04, 3 node ES cluster, 1 instance for Kibana. ES is proxied behind an AWS ELB.
Hard to say really, a lot has changed from 2.x to 6.x. But one major source of slow queries does come to mind.
There was a change in how numerics are stored internally, to use a new datastructure. This datastructure is superior for numeric-style queries (ranges, etc). But it is slow to do single point lookups.
People get into trouble when they have an identifier which is stored as a numeric value (12345), and then query that ID with term/terms style queries (e.g. "find me this specific ID or IDs).
If you have that situation, you should remap your IDs as keywords which are optimized for single term lookups.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.