What elasticsearch does when restarting a cluster?

[yishain11]

I have en elasticsearch cluster consisting 3 nodes, and a pretty big index (~1500 fields), and when it restarts, ether from a "clean" restart or forced shotdown, it takes a while to go back up, and it hoggs the JVM heap and indexing slow down, as described in feeds such as:

Garbage Collection blackout

Hi all, I have set up a ELK server (mostly using defaults) on a VM with 2GB of memory and 200GB of disk. It has been been loading about 250 log files from another server using Filebeat. (The files are only about 300KB each.) Performance has been really terrible, though, and log files are showing numerous errors. Also -- (and the reason for this message) -- I've been seeing extended time periods where the ES server appears to almost completely black out while doing GC. Notice the logs below, w…

ES5 Indexing performance (seems slow)

Hi, When indexing 200000 documents in batches of 1000 using the bulk REST API I am seeing the following log messages: [2016-11-03T21:30:55,263][INFO ][o.e.i.IndexingMemoryController] [node-1] now throttling indexing for shard [[myindex][1]]: segment writing can't keep up [2016-11-03T21:30:57,708][INFO ][o.e.i.IndexingMemoryController] [node-1] now throttling indexing for shard [[myindex][2]]: segment writing can't keep up [2016-11-03T21:31:01,213][INFO ][o.e.i.IndexingMemoryController] [nod…

My question is not how to handle this issues, but to understand: What does elasticsearch tries to do when the cluser is restrating? what is happening behind the scenes that cause this slowdown and heavy memory usage?
To what configurations does it relate to?
I need to be able to explain the process to my peers and then think how to make the cluser restart not stop other process in the machine that also uses JVM heap memory.
Thanks!

[DavidTurner]

@yishain11

What does elasticsearch tries to do when the cluser is restrating? what is happening behind the scenes that cause this slowdown and heavy memory usage?

Elasticsearch is preparing everything it needs to serve searches and indexing requests, loading various data structures from disk and doing other preparatory work.

The threads you linked were nothing to do with startup, so I don't see how they're relevant here.

@yishain11

how to make the cluser restart not stop other process in the machine that also uses JVM heap memory.

Elasticsearch runs in a JVM on its own. It does not share a heap with any other process.

However, let me turn the question around: why does the startup performance matter to you? A properly configured 3-node cluster only needs to restart from cold after a serious disaster affecting multiple nodes.

[yishain11]

Thanks for your answer.

Elasticsearch is preparing everything it needs to serve searches and indexing requests, loading 
various data structures from disk and doing other preparatory work

What preperations? reindexing? remapping?
Does it matter if the data currently stored in elastic is big in size/mapping?

I have two java process in my VM, one of them is elastic. When I push a large number of files to elastic (meaning: a large number of files (~1,000,000 at a time for a while) with a large number of fields), and I stop elastic an restart it to see what happens if the machine would crash, the second process running stop to a crawl (or even carshes completely) and elasticsearch log tells me that:

1. now throttling indexing for shard segment writing can't keep up stop throttling indexing for shards
2. JVM garbage collector reports on overhead, like in the post I linked too (I know that the posts themselves are not related to cluster restart, but the log messages are the same for me, when I restart my cluster).
3. etc

Again, I'm not here to understad the perticular issues I face when I restart, I'm Just trying to understand what elasticsearch is going through when restarting, how many tasks does it has, and how the number of docs or fields in mapping effect the restart.

[DavidTurner]

@yishain11

What preperations? reindexing? remapping?

No, it does not do any reindexing unless you ask it to. I do not know what you mean by remapping.

@yishain11

Does it matter if the data currently stored in elastic is big in size/mapping?

Yes, I would expect it to take longer to restart if it has more data and/or if the data are more complex.

You can find out what the Elasticsearch process is doing at any time using the hot threads API or by taking a thread dump with jstack. You can also increase the logging level to get more information in the logs.

I would expect a certain amount of GC activity when starting up a node - it's working as hard as it can trying to get the node back up as quickly as possible, so it will try and use all the resources that you've allowed it to use. This includes CPU, memory and I/O bandwidth. If it's a problem that it's affecting other processes then it sounds like you will need to isolate these processes better.

[yishain11]

Awsome.
Thank you for your time and patience!