"unable to create new native thread" with default config

Elastic Stack Elasticsearch
1

I'm using an older version of Elasticsearch (1.4.1) to run AtoM (https://www.accesstomemory.org) hosted with Webfaction (1GB ram plan). I'm not running out of memory on my plan (otherwise they just terminate the process) but after navigating around the webapp 2-3 clicks elastic search process throws an error:

Failed to execute [org.elasticsearch.action.search.SearchRequest@8d3584] while moving to second phase
java.lang.OutOfMemoryError: unable to create new native thread

Java version is 1.8 and ulimit -u is fixed to 300.

I've been googling for 2 days now and everyone seem to propose to increase the ulimit to a larger number, other make it unlimited. This is not possible with the hosting provider.
I also contact the hosting support by the response was very vague, saying that Elasticsearch process is spawning alot of threads and basically proposing to play around with:

-Xmx sets the maximum memory heap size
-Xms sets the minimum memory heap size.

which I'm not sure how to do..

What is the best way to deal with inability to spawn more threads? Somehow reduce the number of available ones? The site in question is available here: http://www.lmr-rml.ch and most times it is unavailable because of the 500 error because of the elastic search throwing errors..

Any suggestions?

2

The hosting support answer is dubious at best. Java heap memory can in no way influence thread limits.

Can you please show the full stack trace of the OOM exception and a thread dump with the tool jstack of a running ES process? jstack reveals how many threads are spawned.

How many CPU cores does your machine have?

You may want to tweak Elasticsearch thread pool settings in the configuration in order to reduce the resource consumption and power of ES, but that is not very elegant.

If that is not convenient, you may want to try latest Elasticsearch 2+ but without AtoM to find out if Webfaction restrictions allow ES to run OOTB. Elasticsearch 2+ has many improvements in that area.

If so, you should encourage AtoM to migrate to Elasticsearch 2+ https://www.accesstomemory.org/en/docs/2.3/admin-manual/installation/requirements/#installation-requirements

If you can not run ES OOTB you might want to find another hosting provider.

3

@jprante, thank you so much for your response!

I'm not sure how many cores the machine has (it's a shared hosting server, with and SSD hard drive and 1 GB of RAM). I also know it is a 64-bit machine. Is there any way I can check myself how many cores are available?

I would gladly use the latest software available, but unfortunately AtoM does not support version 2.0+ (https://www.accesstomemory.org/en/docs/2.2/admin-manual/installation/requirements/#installation-requirements).

Full error stack looks like this, and I'm getting of a lot of them for every request I'm sending to the server.

EDIT: Discuss wouldn't let me paste the whole stack here, saying I'm over the characters limit, so I'm attaching it in a text file available here

How would I got about getting thread pool settings (I noticed there are 2 ways of doing it: via settings file and http rest request. I prefer the former, since, ideally, I'd launch the service and forget about it. What settings should I tweet and what values would you suggest to try out. Real life examples would be awesome!

4

To find out the number of usable CPU cores, try to run lscpu

Or you can try to run this Java program AvailableProcessors.java

class AvailableProcessors {
  public static void main(String[] args) {
    System.out.println(java.lang.Runtime.getRuntime().availableProcessors());
  }
}

by executing

javac AvailableProcessors.java
java AvailableProcessors

You can only configure thread pools in old ES versions (like 1.x) from configuration file.

Refer to this page

https://www.elastic.co/guide/en/elasticsearch/reference/1.7/modules-threadpool.html

You can change all thread pools you want to minimal settings with a fixed thread pool, like this

threadpool:
    search:
        type: fixed
        size: 4
        queue_size: 100

where size is the number of threads which may run concurrently. You can even use size: 1 but the impact on performance is severe.

You should also ensure if your hosting provider counts Java threads as processes, which makes things even worse . Maybe a hard process limit applies, making the product of this hosting provider barely usable with any Java application.

5

Jörg, you're my hero!

I ran lscpu

this is the response:

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 40 On-line CPU(s) list: 0-39 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Stepping: 2 CPU MHz: 1200.000 BogoMIPS: 4607.27 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 25600K NUMA node0 CPU(s): 0-9,20-29 NUMA node1 CPU(s): 10-19,30-39

I'm not really sure what is the limit of threads allowed for this kind of configuration, but I used the values you provided to configure elasticsearch and they worked! No more crashes! (at least for now).

I'll monitor the performance and will try to play around these values to see how it plays out.
Thank you again! You've been of a great help to me.

6

With CPU(s): 40, it seems Elasticsearch will select the maximum settings available.

It seems that in your case, Java picks up the physical cores and not the one you assume for the VM. Maybe this is even correct, in the case a VM does not have CPU restrictions configured.

ES also provides a handy abbreviation, the processors setting, maybe you already read it in the docs:

https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-threadpool.html#processors

This will scale down/up all dynamic thread pools to a configured core count, overriding the detected core count.