ELK to slow

Elastic Stack Logstash
#1

I have elk installation:
logshash -> rmq -> logstash(2 instances) -> Elasticsearch
Elasticsearch cluster with 3 nodes. elasticsearch-5.2 (8 cores, 32 Gb, SSD) 12Gb HEAP
"number_of_replicas": 0

cluster.name: logs
node.name: el1,2,3
network.host: 0.0.0.0
transport.host: 172.30.30.180
http.port: 9200
indices.recovery.max_bytes_per_sec: 150mb
http.cors.enabled: true
http.cors.allow-origin: "*"
discovery.zen.ping.unicast.hosts: ["192.168.0.1", "192.168.0.2", "192.168.0.3" ]
discovery.zen.minimum_master_nodes: 2
action.destructive_requires_name: true

Logstash-5.2

pipeline.workers: 8
pipeline.batch.size: 500
-Xms4g
-Xmx4g

And my cluster can't consume more than 1.5 k events per sec from rmq.
My application produce more than 2.5k events per sec.
Almost all events are logs in json format so logstash config looks like this:

filter { json { source => "message" } }

I believe my cluster can work better. How cat I find a bottleneck? LA on severs about 0.2

#2

Have you tried using the json_lines codec in the input instead of using the json filter?

#3

Actually I think issue in Elastic configuration.
I tried to remove all filters, left only input and output and result almost the same - 1,7-1,8k events.

#4

You have a 12G heap on each Elasticsearch node?

How many indices do you have? How many shards per index do you have?

#5

Only 1 logstash index, cluster is almost emty.
Shards 5.

#6

If I set "number_of_replicas": 0 indexing speed increase to 10-12K

#7

What is the average size of your events?

#8

This does suggest your cluster is at least somewhat I/O bound. Removing replicas takes away 1/2 of the writes if you have 1 replica per primary shard. However, it seems you have a 5x speed boost. I'm with @Christian_Dahlqvist, here, in wondering how large your documents/event are.

#9

~ 1.4 KB

#10

The increase in performance due to disabling replicas is larger than I would expect. How many replicas did you have configured before? Apart from disk I/O, network usage should also drop with reduced number of replicas. What type of networking do you have in place? Is the cluster deployed on bare-metal hardware or VMs with shared networking?

#11

Bare metal, 1000Mb/s network.

'{ "number_of_replicas": 0 }' - > index.rate: 8010.0
'{ "number_of_replicas": 1 }' - > index.rate: 2048.2

#12

Is this the raw size or the size of the JSON documents you are ingesting? How many fields? Do you use nested mappings?

Do you have X-Pack monitoring installed so you can provide graphs around node metrics?

#13

1.4 RAW size.
~ 40 fields.

Yes, I have X-Pack, what graphs can help you?

#14

What is the average size of the documents once converted to JSON (I find this a better measure than raw size as it accounts for any enrichment being performed)?

Graphs showing CPU and heap usage as well as indexing and query rates during indexing would be useful. Full screenshots of the Nodes/Overview and Nodes/Advanced screens would give us a better idea.

#15

1569 bytes, I count it like this: "store"."size_in_bytes"/"docs"."count"
Graphs, now {"number_of_replicas": 0 } :

ES_Overview.png1715×820 117 KB
ES_Advanced.png1717×1516 253 KB

#16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.