Elastic search Tuning Performance

Elastic Stack Elasticsearch
1

I have 2 servers with same hardware on both:
cpu
16gb Ram
Virtual Storage 500gb
Unix System

I tried to index a 250mb file on one of them using logstash, it took 30 mins to index which is a really poor performance. Note I am using default settings in all configurations only changed JVM Heap to 8gb on each server. I installed metric beats to view the metrics on the server and they give

02
02.PNG1714×883 113 KB

and after like 15mins it became

SnipImage
SnipImage.jpg1664×754 170 KB

Can you please tell me what to edit and follow and I already read the tuning speed guidelines but I got lost so I really need help in choosing the setting and which to change. Thank you in advanced

2

It seems like you have swap enable and are using some of it. We do recommend disabling swap when running Elasticsearch.

How many indices and shards are you concurrently indexing into?

How many concurrent indexing threads/processes are you using?

3

i am running now 3 pipelines on logstash and they are behaving with same speed as running one pipeline , and i am indexing them to the same index and 1 shard only, and i forgot to mention i made replicas to 0

4

If you are indexing into a single index with only one primary shard and no replicas only one node will actually be indexing. If you change the number of primary shards to 2 so each node can get one I would expect you to see improved throughput.

If you are running Logstash on the same host as Elasticsearch, it would be interesting to see how CPU usage breaks down between these two processes.

5

i made 1 node master and the other data so if I removed the master and made it 2 shards it will improve more and i am running logstash on the data node now

6

If the nodes have the same specification, make them both data nodes. You can still keep just one node as master eligible. Performance issues are also often easier to troubleshoot of Logstash runs on a separate host.

7

okay, and i read about resfresh intervals and node buffers any setting that can improve indexing speed

8

If indexing takes too long but Elasticsearch seems alright, maybe Logstash is on the slow side and you need to increase the amount of workers in your configfile.
Check your config with
curl -XGET 'localhost:9600/_node/pipelines?pretty'
and look for workers.
If workers is just one look here https://www.elastic.co/guide/en/logstash/6.6/tuning-logstash.html
and increase them.

Good Luck!

9

i tried to increase the number of workers and it gave same results as before

10

What is the full output of the cluster stats API and the node stats API For the data node?

11

Alright, what kind of data do you import? Does it contain a lot of rows. Is it a Json file? What does your logstash processing pipeline look like INPUT - FILTER - OUPUT?

12

its a csv file with a lot of columns and yeah it has a lot of rows
here is the logstash config

input {
file {
path => "C:/Users/Fadi/Desktop/internship/procera_Justice-*.csv"
start_position => "beginning"

}
}
filter {
csv {
separator => "|"
columns =>
["egressinterface", "ingressinterface", "applicationid", "destinationipaddress"
, "destinationtransportport", "sourcetransportport","flowstartmilliseconds","flowendmilliseconds","flowstartseconds","flowendseconds","procerainternalrtt","proceraexternalrtt","proceraservice",
"proceraserverhostname","proceraincomingoctets","proceraoutgoingoctets","octetdeltacount","proceraqoeincomingexternal"
,"proceraqoeincominginternal","proceraqoeoutgoingexternal","proceraqoeoutgoinginternal","sourceipaddress","procerasubscriberidentifier","proceraggsn","procerasgsn",
"procerarnc","proceraapn","procerauserlocationinformation"
,"procerarat","observationpointid","proceradeviceid","proceraimsi","chargingid","ProceraServiceObject","protocol_identifier","procerahttpresponsestatus","procerahttpuseragent","procerahttpurl"]
convert => {

"egressinterface" => "integer"
"ingressinterface" => "integer"
"applicationid" => "integer"
"destinationtransportport" => "integer"
"sourcetransportport" => "integer"
"procerainternalrtt" => "integer"
"proceraexternalrtt" => "integer"
"proceraincomingoctets" => "integer"
"proceraoutgoingoctets" => "integer"
"octetdeltacount" => "integer"
"proceraqoeincomingexternal" => "integer"
"proceraqoeincominginternal" => "integer"
"proceraqoeoutgoingexternal" => "integer"
"proceraqoeoutgoinginternal" => "integer"
"observationpointid" => "integer"
"chargingid" => "integer"
"protocol_identifier" => "integer"
"procerahttpresponsestatus" => "integer"

}

}

date {
match => ["flowstartseconds", "yyyy-M-d HH:mm:ss", "yyyy-M-dd HH:mm:ss", "yyyy-MM-d HH:m:ss", "yyyy-MM-dd HH:mm:ss"]
target => "flowstartseconds"
locale => "en"
timezone => "UTC"

}
date {
match => ["flowendseconds", "yyyy-M-d HH:mm:ss", "yyyy-M-dd HH:mm:ss", "yyyy-MM-d HH:mm:ss", "yyyy-MM-dd HH:mm:ss"]
target => "flowendseconds"
locale => "en"
timezone => "UTC"

}

}
output {
elasticsearch {
hosts => "localhost"
index => "procera"
document_type => "data"

}
stdout {}
}

13

this is the cluster api
{
"_nodes" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"cluster_name" : "elastic",
"cluster_uuid" : "d7Yg9IShRC6Q9-NbDAv1bw",
"timestamp" : 1548938459707,
"status" : "green",
"indices" : {
"count" : 27,
"shards" : {
"total" : 27,
"primaries" : 27,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"primaries" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
},
"docs" : {
"count" : 6816580,
"deleted" : 1013
},
"store" : {
"size" : "4.5gb",
"size_in_bytes" : 4908194710
},
"fielddata" : {
"memory_size" : "32.4kb",
"memory_size_in_bytes" : 33256,
"evictions" : 0
},
"query_cache" : {
"memory_size" : "528kb",
"memory_size_in_bytes" : 540737,
"total_count" : 56974,
"hit_count" : 6179,
"miss_count" : 50795,
"cache_size" : 517,
"cache_count" : 722,
"evictions" : 205
},
"completion" : {
"size" : "0b",
"size_in_bytes" : 0
},
"segments" : {
"count" : 244,
"memory" : "11.8mb",
"memory_in_bytes" : 12373717,
"terms_memory" : "7.6mb",
"terms_memory_in_bytes" : 8025412,
"stored_fields_memory" : "1.7mb",
"stored_fields_memory_in_bytes" : 1848112,
"term_vectors_memory" : "0b",
"term_vectors_memory_in_bytes" : 0,
"norms_memory" : "199.3kb",
"norms_memory_in_bytes" : 204096,
"points_memory" : "784kb",
"points_memory_in_bytes" : 802865,
"doc_values_memory" : "1.4mb",
"doc_values_memory_in_bytes" : 1493232,
"index_writer_memory" : "16.3mb",
"index_writer_memory_in_bytes" : 17192532,
"version_map_memory" : "520b",
"version_map_memory_in_bytes" : 520,
"fixed_bit_set" : "288.1kb",
"fixed_bit_set_memory_in_bytes" : 295024,
"max_unsafe_auto_id_timestamp" : 1548927891218,
"file_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 2,
"data" : 1,
"coordinating_only" : 0,
"master" : 1,
"ingest" : 2
},
"versions" : [
"6.5.4"
],
"os" : {
"available_processors" : 8,
"allocated_processors" : 8,
"names" : [
{
"name" : "Linux",
"count" : 2
}
],
"mem" : {
"total" : "31gb",
"total_in_bytes" : 33315569664,
"free" : "1.4gb",
"free_in_bytes" : 1608933376,
"used" : "29.5gb",
"used_in_bytes" : 31706636288,
"free_percent" : 5,
"used_percent" : 95
}
},
"process" : {
"cpu" : {
"percent" : 9
},
"open_file_descriptors" : {
"min" : 296,
"max" : 414,
"avg" : 355
}
},
"jvm" : {
"max_uptime" : "3.8h",
"max_uptime_in_millis" : 14019209,
"versions" : [
{
"version" : "1.8.0_161",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "25.161-b14",
"vm_vendor" : "Oracle Corporation",
"count" : 2
}
],
"mem" : {
"heap_used" : "2.6gb",
"heap_used_in_bytes" : 2883739064,
"heap_max" : "15.9gb",
"heap_max_in_bytes" : 17110138880
},
"threads" : 128
},
"fs" : {
"total" : "982.3gb",
"total_in_bytes" : 1054755577856,
"free" : "907.5gb",
"free_in_bytes" : 974524428288,
"available" : "865.6gb",
"available_in_bytes" : 929490112512
},
"plugins" : ,
"network_types" : {
"transport_types" : {
"security4" : 2
},
"http_types" : {
"security4" : 2
}
}
}

14

the nodes api is too big to post here

15

when you tested more workers on logstash did you add the line
workers => 20
to your elasticsearch output section? I can't see it currently. I do not mean pipelines, just workers that would go under index => "procera" for example

Can you monitor your network throughput of logstash's port with tcpdump, dstat or something you are familiar with? Your Logstash surely has quite a bit to do. Maybe you can also check the processes with top, htop, etc. and see what's using how much on your system.

Your dataflow seems to me CSV File - > Logastash -> Elasticsearch
One of these is the bottleneck, maybe you can split the file into smaller chunks of 50MB each?

16

i just added 10 workers to the pipeline of procera in pipelines.yml file the default was 4, and i will check this network throughput plus i cant split the file because my final result every 1 hour I receive 4 of this file and I want to index them in a really fast manner

17

cpu
this is the top command

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.