ES Linux vs Windows - 100% mem use during index, different search times (Linux is slower)

Elastic Stack Elasticsearch
1

We have been running a Windows ES cluster for some time. We decided that it seemed Linux often was considered faster in performance... So when we upgraded our cluster, we used Linux boxes.

A little background on the systems:
Both are running on the same Hyper-V infrastructure.
Both clusters are 20GB RAM 4vCPU boxes using the same shared storage.

Windows cluster was ES 2.1
Linux cluster is ES 2.3

When indexing on the Linux cluster, we notice our system monitors alerting because they go to nearly 0% free memory in some cases... this seems to become more frequent with each reindex (we reindex data a lot, multiple times per hour).

At the moment, the Linux cluster is in pre-production testing, no ongoing queries against it.
The Windows cluster however hovers (according to Task Manager) at about 13GB of 20GB used.

Why does ES on Linux seem to cause 100% memory utilization.

Further, we also notice in testing that the time taken reported in a query is longer for the Linux cluster (which currently has no load) than the Windows cluster (which is in production).

With multiple queries on the Windows cluster, the time taken will be between 1-10, whereas on the Linux cluster the time taken always reports between 13-45.

Note that we are using the same data for both tests and the same query... the only difference is the cluster. To get the lowest time results, we execute the same query in rapid succession. I expect a time of 1 is because the result is cached... shouldn't the same happen on the Linux cluster?

For Linux, we do have memlock on and confirmed working. We do have the heap set to 10GB (50% of total RAM).

Is there perhaps something I'm not properly tuning on Linux? I expected Linux to perform better than the Windows version.

Thanks!
Heath

2

Are you talking overall system memory, or heap?

3

What Linux distribution and version is this? What kernel? What filesystem? What network setup? What do you mean by "same Hyper-V infrastructure"? And what is "shared storage" - SAN? NAS? NFS? What about swap space?

What Java distribution do you use?

Why do you use different ES versions and compare them? It's like apples and oranges.

1 Like
4

Overall memory. Perhaps just reporting differently between Windows and Linux?

5

Jprante: Ununtu 14.04, Java 8 (most recent version)

The rest of the infrastructure is identical. So rather than deal with specs of what could be better/different, the question is. A) is there an expected performance decrease from ES 2.1 to 2.3? Or B) is there an expected performance/memory usage difference between Windows and Linux installs?

6

Here you mentioned the heap as 20 GB and [quote="haldrich, post:1, topic:54582"]
For Linux, we do have memlock on and confirmed working. We do have the heap set to 10GB (50% of total RAM).
[/quote]

Here you are saying that the heap as 10 GB.

Hope here is the difference

7

Your question ignores that you must take careful analysis before addressing ES as a possible cause.

Also, the assumption there is one "Windows" and one "Linux" is ignoring the fact that there are different distributions, versions, and configurations with strong influence on performance. It's simply not a matter of "increase" or "decrease" but more a matter of configuration.

Of course there are always different performance and memory usages, but ES is not the cause in most cases.

8

The heap itself is 10GB. Total ram is 20

9

Thanks but that doesn't go to the question.
Performance is different on the same exact hardware setup.

The question is not in analyzing performance, but what would be likely responsible for the difference.

Since the hardware is identical, it isn't that.

So is it a Windows Linux thing or an ES 2.1 vs 2.3 thing?

10

That's the OS caching things, see https://www.elastic.co/guide/en/elasticsearch/guide/2.x/heap-sizing.html#_give_less_than_half_your_memory_to_lucene

It's totally normal.