I would like to add this... under the hood, each shard is a Lucene index resides in its own directory and each index can hold up to 2 billion documents. When you have this many shards, you'll see this many directories created under the hood. I've ingested 2B into a shard before b/c I just wanted to see how it handled indexing and searching. Each shard seems to response very well when it has upto 1B documents (good for indexing, searching, loading, etc) When ES is up and the index is active, in theory, it will "talk" to all the shards, thus it will consume a significant amount of system resources. That's why you'll see the yellow b/c it's still in the process of "talking to all the shards"
Based on what you said above, 12 CPU and 16GB RAM. The system RAM is low b/c you can have the HEAP_SIZE set to 30GB max so think about this resource if you can have the opportunity to upgrade this machine.
I suggest to do a little bit of planning in terms of
- how many documents (not data size in terms of GB) are you going to index per day?
- how many documents are you planning to keep in one index?
- overtime, how many documents or indices do you want to keep around?
Depending on what you have, you'll decide how many shards you should have per index and how many indices you should create and maintain in this machine. For testing, I understand you can create as many indices with as many shards as you want but eventually the system will not be able to handle the load effectively (as you've seen), ES is not a problem here, it's the "limited resources" given to ES.