Slow performance after upgrade from 2.4

Elastic Stack Elasticsearch
1

Hey all,

I recently deployed a new logging cluster on Kibana / ES 6.2.4
My old cluster was on 2.4

Performance on the new cluster is terribly slow. Queries that the 2.4 cluster is returning in a matter of 3-5 seconds are taking > 30 seconds (which is the default timeout) on my 6.2 cluster.

I am wondering if there are any major gotchas that I haven't considered while upgrading from 2.4 to 6.

In the browser, I see the _msearch network request either 502 or get cancelled.
It is using http/1.1 and no SSL. But my understanding is that ES doesn't support http/2, so would having an http/2 proxy in front of it yield performance improvements?

This is on a new cluster, it's receiving fresh data from fluent / td-agent -- I never tried to do a data dump from ES 2.4 to 6.2

Any help would be appreciated, I understand this is a bit vague of a description, I'm happy to provide more information!

This is running in Ubuntu 16.04, 3 node ES cluster, 1 instance for Kibana. ES is proxied behind an AWS ELB.

Thanks!

2

Hard to say really, a lot has changed from 2.x to 6.x. But one major source of slow queries does come to mind.

There was a change in how numerics are stored internally, to use a new datastructure. This datastructure is superior for numeric-style queries (ranges, etc). But it is slow to do single point lookups.

People get into trouble when they have an identifier which is stored as a numeric value (12345), and then query that ID with term/terms style queries (e.g. "find me this specific ID or IDs).

If you have that situation, you should remap your IDs as keywords which are optimized for single term lookups.

I wrote some more about it here: The queries for numeric fields are slower after upgraded the cluster from 2.4.5 to 5.6.3 - #2 by polyfractal

Unsure if that's the situation you're running into, but it's a common cause of slower queries after upgrading.