Got one question. Working on my elastic stack. Basically it's "developing production".
Got one server with 16GB of RAM, 4 CPUs, ELK 5.4.0. No cluster or extra nodes.
Got no problems with adding data and searching, the problems gets with dashboards with multi visualizations.
I figure, the problem is probably CPU?
With development, also the document size increased as I try to automate as much as possible. So indexes are created daily with around 5-6 mio documents per index. So with few months of data I'm already at several 100 millions of documents.
So when I open a dashboard with multiple visualisation I receive a timeout error in kibana and elasticsearch log says:
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService$7@59b0d1f1 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@395e3b7a[Running, pool size = 7, active threads = 7, queued tasks = 1000, completed tasks = 3105230]]
So i'm thinking of maybe increase the search queue_size. Would that even make sense? What would be the optimal size in my situation? Best would be probably to add two more nodes and create a cluster.