Elastic Tuning

Elastic Stack Elasticsearch
1

Hello Elastic Team,

I am new in elastic search and i want to ask ,

I have 3 servers with 16 gb ram and virtual storage and 4 CPUs on each which is intel xeon E5-2699 v4 2.2 ghz 4 cores on each.

okay so i want to index documents through logstash and if you can tell me the instructions i can do to tune logstash and elasticsearch. My document is a csv file with about 150 columns of size 8gb. IT took 3 days to index it which is a really poor performance since this file is received every hour. I have all settings on default nothing touched in elastic configuration or logstash. Can you tell me if i need more servers and what type of nodes should i assign to each server and what to configure each server on logstash and elastic because i need maximum index performance. Thank you.

2

Make sure you have followed these tuning guidelines for indexing for Elasticsearch. Also make sure you have optimised your mappings so you reduce the amount of unnecessary work.

When it comes to Logstash you should have a look at these guidelines. As you are parsing large CSV files, it is worth noting that the dissect filter can be significantly faster than the csv and/or grok filters.

It is also worth having a look at the cluster while you are indexing to try and see what resource is limiting performance. For high indexing rates it is often CPU, network or disk performance (since you are using virtual storage I would recommend looking at disk I/O and iowait).

3

okay and what can i do if i have poor i/o and disk rate

4

Get faster storage or scale out the cluster (unless storage is likely to still be the bottleneck).

5

Use SSDs. If you are already using it, see what's the output of iostat.

6

this is my io io this is it and how can i scale out the cluster?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.